CodeVetter · Repo Unpack · Data flow
firecrawl/firecrawl
Inbound JSON is validated by Zod schemas in controllers/v2/types.ts, then the request is stamped with team/org identity (ACUC), credits are checked against Autumn, and the job is either executed inline (sync scrape) or persisted as a NuQ row. The scrapeURL pipeline rewrites/permutes the URL, consults the blocklist, robots.txt, safe-mode and threat-protection policy, then walks a quality-ordered engine waterfall; each engine returns raw HTML which is normalized (removeUnwantedElements), converted to markdown (Go native lib or HTTP service), and passed through transformers (LLM extract, JSON, screenshots, change tracking) and postprocessors. Results are persisted to Bigtable job state, indexed to GCS+Postgres, billed per-feature, logged to PubSub/ClickHouse, and returned synchronously or delivered via webhook.
- Request validation is schema-first: scrapeRequestSchema.parse(req.body) runs inside an 'api.scrape.validate' OTel span, and ZodError is mapped centrally to a 400 with a strict 'Unrecognized key' message for v2.
- authenticateUser resolves the API key to an ACUC (auth credit usage chunk) via Redis-cached DB RPCs (authCreditUsageChunk), supports PREVIEW_TOKEN bypass, USE_DB_AUTHENTICATION mode, OAuth token introspection and MCP delegated credentials; the chunk carries team_id, org_id and feature flags used everywhere downstream.
- Credits are enforced by Autumn: checkCreditsMiddleware calls autumnService.checkCredits, fails open when Autumn is unreachable, clamps crawl/batch limits to remaining balance, and returns 402 with pricing links when denied; scrape-worker bills actual usage post-hoc via billScrapeJob → billTeam.
- Sync scrapes run a NuQJob in-process: scrapeController builds a NuQJob<ScrapeJobData> with skipNuq:true and calls processJobInternal directly inside teamConcurrencySemaphore.withSemaphore, so the queue row is skipped but the same worker code path executes.
- Async jobs persist as rows in nuq.queue_scrape via scrapeQueue.addJob/addJobs (queue-jobs.ts), with routing through nuq-router/scrapeQueueFdb when the FDB backend is selected; workers consume via runNuqWorker → processJobInternal.
- Inside scrapeURL, URL handling precedes engines: safe-mode resolution strips credentials/login actions, threat-protection checkUrl runs before any fetch with a per-scrape dedup map, robots.txt is consulted via shouldCheckRobots/fetchRobotsTxt, and urlSpecificParams/rewriteUrl normalize the target.
- buildFallbackList computes the engine waterfall: exchange/index short-circuits first, then engines are filtered by required FeatureFlags and sorted by quality (index=1000, wikipedia=500, fire-engine chrome-cdp=50, playwright=20, tlsclient=10, fetch=5, file engines=-20); scrapeURLLoopIter runs each and deems success on markdown length + status code + absence of page error.
- Engine output flows through htmlTransform (removeUnwantedElements), parseMarkdown, executeTransformers (llmExtract, deterministicJson, diff/changeTracking, audio/video, menu, product, query, sendToSearchIndex) and postprocessors (youtube) before becoming the Document returned to callers.
- Crawl state lives in Redis: StoredCrawl, per-URL locks (lockURL/lockURLsIndividually), job membership (addCrawlJob/addCrawlJobDone), robots.txt cache and crawl errors are managed by lib/crawl-redis.ts; completion fans out through crawlFinishedQueue to finishCrawlSuper.
- Terminal job state is written to Bigtable (job-state-store.ts, salted uuidv7 row keys) so scrape-status can read results after Redis TTLs expire; a SCRAPE_STATE_BARRIER_MS=2000 wait in scrape-worker lets the write land before sync responses return.
- The URL index is a two-tier cache: metadata/verdicts in Postgres via db/rpc.ts functions (queryIndexAtSplitLevel, queryEngpickerVerdict) plus document blobs in GCS (getIndexFromGCS), with a Redis index-cache layer and screenshot URL re-signing.
- Webhooks are queued and delivered asynchronously: createWebhookSender builds per-crawl senders, delivery.ts filters events and persists deliveries, and shutdownWebhookQueue is part of the graceful-exit chain in index.ts.
- Request/audit logging is pipelined: logRequest/logScrape in services/logging/log_job.ts emit to PubSub/GCS, cclog-worker handles ClickHouse logs, and SIEM events go through lib/siem-logging to per-team transports (e.g. Azure Sentinel) via a RabbitMQ-delivered worker.