CodeVetter · Repo Unpack · Feature catalog

firecrawl/firecrawl

Scanned commit 556c12c9d2 dated 2026-09-21 — last upstream commit. Generated by codevetter codevetter 1.12.0 on 2026-09-22. Deep analysis by swe-2 via the devin CLI on 2026-09-22 (runtime not recorded).

The v2 API (routes/v2.ts) is the primary surface: scrape, batch scrape, crawl (+status/errors/cancel/websocket), map, search (+developer/research proxies), extract/agent jobs, parse (multipart upload), browser/interact sessions, monitors, Slack integration, team usage/SIEM/threat-protection administration, and keyless free-tier endpoints. v1 mirrors the core endpoints (scrape/crawl/map/extract/search/deep-research/llms.txt) and v0 is a thin deprecated shim. Every mutating endpoint passes through the same middleware chain: authMiddleware (API key, OAuth, MCP delegated credential, or keyless), countryCheck, checkCreditsMiddleware (Autumn), blocklistMiddleware, and optionally idempotencyMiddleware.

POST /v2/scrape runs scrapeController: zod-validates via scrapeRequestSchema, resolves safe mode and threat-protection policy, checks permissions and per-key format restrictions, acquires a team concurrency semaphore slot, then executes the scrape in-process via processJobInternal (skipNuq: true) rather than enqueueing.
apps/api/src/routes/v2.ts#L235-242 apps/api/src/controllers/v2/scrape.ts#L57-99 apps/api/src/controllers/v2/scrape.ts#L358-441
POST /v2/crawl creates a StoredCrawl in Redis (saveCrawl), registers a NuQ job group for concurrency/delay control, optionally expands a natural-language prompt into crawler options via generateCrawlerOptionsFromPrompt, then enqueues a mode:'kickoff' job that discovers links and fans out child scrape jobs.
apps/api/src/controllers/v2/crawl.ts#L218-267 apps/api/src/controllers/v2/crawl.ts#L324-400 apps/api/src/services/worker/scrape-worker.ts#L1818-1840
Crawl lifecycle endpoints: GET/DELETE /v2/crawl/:jobId (status via crawlStatusController, cancel via crawlCancelController), a WebSocket status channel on the same path, /crawl/ongoing|active, /crawl/:jobId/errors, /batch/scrape/:jobId reusing crawlStatusController, and /crawl/params-preview.
apps/api/src/routes/v2.ts#L282-366
POST /v2/map and /v2/search are credit-gated discovery endpoints; search chains providers fire_engine → searxng → DuckDuckGo (search/index.ts), and /v2/search/:jobId/feedback plus /v2/feedback collect quality signals.
apps/api/src/routes/v2.ts#L189-210 apps/api/src/routes/v2.ts#L274-280 apps/api/src/search/index.ts#L29-60
/v2/extract and /v2/agent are async job APIs (POST to start, GET /:jobId for status, agent also has trace/skill/snapshots/thread/cancel endpoints); both are billed at 20 credits via checkCreditsMiddleware(20) and extract carries a deprecationMiddleware('v2_extract') notice.
apps/api/src/routes/v2.ts#L368-441
Browser sessions are a first-class feature: POST/GET/DELETE /v2/browser (aliased /interact), /:sessionId/execute, /:sessionId/replay[/:pageId], and a public /browser/webhook/destroyed callback for the browser infra to report teardown.
apps/api/src/routes/v2.ts#L625-666
Monitors (scheduled re-scrapes with alerting) get full CRUD plus /monitor/:id/run, /checks history, and unauthenticated email confirm/unsubscribe endpoints; a scheduler in queue-worker.ts enqueues due checks via services/monitoring/scheduler.ts.
apps/api/src/routes/v2.ts#L533-596 apps/api/src/services/queue-worker.ts#L25-32 apps/api/src/services/monitoring/scheduler.ts
A Slack app is embedded in the API: /v2/slack/oauth/start|callback, /status, /channels, DELETE /installation, plus unauthenticated /slack/commands and /slack/events verified by Slack's request signature (raw body captured by captureRawBody in index.ts).
apps/api/src/routes/v2.ts#L598-623 apps/api/src/index.ts#L72-88
Team administration endpoints expose credit/token usage (current + historical), concurrency-check, queue-status, activity, threat-protection config incl. Zscaler test/sync/categories, and SIEM logging config with a test delivery route.
apps/api/src/routes/v2.ts#L443-531 apps/api/src/controllers/v2/team-threat-protection.ts apps/api/src/controllers/v2/team-siem-logging.ts
/v2/parse accepts multipart file uploads (multer, 50MB cap, memoryStorage) or an uploadRef pointing at a prior PUT /v2/parse/upload/:uploadId body, converting documents to markdown via the same scrapeURL pipeline engines.
apps/api/src/routes/v2.ts#L123-233
A keyless free tier allows unauthenticated scrape/search/interact/parse requests capped per IPv4 per day when KEYLESS_CREDITS_PER_DAY and KEYLESS_REQUESTS_PER_DAY are both set; the trusted-proxy path uses x-firecrawl-keyless-ip authenticated by KEYLESS_PROXY_SECRET, with optional Spur VPN/TOR screening.
apps/api/src/controllers/auth.ts#L466-630 apps/api/src/config.ts#L101-121
v0 and v1 routers remain mounted: v0 exposes /v0/crawl, /v0/scrape, /v0/keyAuth and health endpoints behind deprecationMiddleware; v1 mirrors scrape/crawl/map/extract/search plus deep-research and generate-llmstxt status endpoints.
apps/api/src/routes/v0.ts#L14-43 apps/api/src/routes/v1.ts apps/api/src/index.ts#L133-135
Research/search proxying is conditionally mounted: /v2/search/research, /v2/research (legacy), /v2/search/developer and deprecated /developer only exist when RESEARCH_PROXY_URL is configured, each behind its own RateLimiterMode.
apps/api/src/routes/v2.ts#L680-709