CodeVetter · Repo Unpack · Behavior traces
firecrawl/firecrawl
Three flows were traced end-to-end. Startup: harness.ts supervises containers and child processes, then index.ts initializes blocklist/engine-forcing/exchange catalog and mounts routers. Sync scrape: middleware chain → scrapeController → semaphore → processJobInternal → startWebScraperPipeline → scrapeURL → engine waterfall → transforms → billing/logging → Document. Crawl: crawlController persists Redis crawl + NuQ group, enqueues a kickoff job, and workers discover links, enqueue child single_urls jobs, track completions via addCrawlJobDone and finish via crawl-logic.
- Startup (local dev): `pnpm harness` → tsx src/harness.ts --start → detectContainerRuntime() (docker|podman) → setupNuqPostgres() builds apps/nuq-postgres image, runs container, waits with pg SELECT 1 → setupNuqRabbitMQ() runs rabbitmq:3-management → spawns api (index.ts), worker, N nuq-workers, prefetch/reconciler, extract and index workers via execForward.
- API boot: index.ts loads dotenv+config, installs cacheableLookup on global http/https agents, registers body parsers with rawBody capture, mounts routers, then startServer() awaits initializeBlocklist(), initializeEngineForcing(), warmExchangeCatalog() before app.listen; SIGTERM drains via a 60s Kubernetes wait then nuqShutdown → webhook → indexer → pubsub → tracing shutdown.
- Scrape request, step order: requestTimingMiddleware → authMiddleware(Scrape, allowKeyless) → authenticateUser → countryCheck (geoip on req.ip when actions/headers/agent present) → checkCreditsMiddleware(1) → scrapeBlocklistMiddleware → scrapeController → scrapeRequestSchema.parse → resolveSafeMode → resolveThreatProtection → checkPermissions → checkKeyFormatRestriction → teamConcurrencySemaphore.withSemaphore → processJobInternal.
- Worker execution: processJobInternal wraps the job in zero-data-retention trace context → processJobWithTracing starts a Redis lock-extension interval for non-FDB jobs → dispatches on job.data.mode (kickoff → processKickoffJob, kickoff_sitemap → processKickoffSitemapJob, single_urls → processJob) → processJob races startWebScraperPipeline against remaining timeout → runWebScraper → scrapeURL.
- Engine waterfall: scrapeURL → buildMetaObject → threat checkUrl → buildFallbackList (exchange/index short-circuit, else feature-flag filtered quality sort) → scrapeURLLoop iterates engines calling scrapeURLWithEngine → scrapeURLLoopIter validates output (markdown non-empty, 2xx/304, no page error, auto-escalates stealthProxy on 401/403/429) → executeTransformers + postprocessors → Document.
- Crawl acceptance: crawlController → crawlRequestSchema.parse → safe-mode/threat/permission/key-restriction gates → optional prompt → buildPromptWithWebsiteStructure + generateCrawlerOptionsFromPrompt → merge with explicit options → initializeRequestCredits (Bigtable shards) → crawlToCrawler + getRobotsTxt → resolveNewGroupBackend → crawlGroup.addGroup → saveCrawl → markCrawlActive → _addScrapeJobToBullMQ(mode:'kickoff') → return {id, url}.
- Crawl fan-out (worker side): the kickoff job handler discovers links (WebCrawler/generateURLPermutations, sitemap via scrapeSitemap), checks discovered URLs against threat policy, then addCrawlJobs inserts child single_urls NuQ jobs; each child completion calls addCrawlJobDone and the last one pushes to crawlFinishedQueue handled by finishCrawlSuper.
- Persistence path for a scrape result: pipeline Document → GCS upload when saveScrapeResultToGCS → index write (saveIndexToGCS/upsertCachedIndexEntries) unless ZDR → writeScrapeJobState to Bigtable → billScrapeJob computes credits via calculateCreditsToBeBilled → logScrape → webhook sender emits crawl.page/scrape events → addCrawlJobDone when part of a crawl.
- Keyless request path: authMiddleware with allowKeyless → request lacks Bearer token → keyless branch resolves client IPv4 (or trusted x-firecrawl-keyless-ip) → optional Spur suspicious-IP check → consumeKeylessRequest decrements Redis daily counters → synthetic keylessTeamId(ip) → credits reserved via reserveKeylessCredits in the controller and refunded on failure via adjustKeylessCredits.
- Shutdown: SIGTERM → mcpActionLogRetention.stop() → 60s GCE drain wait under IS_KUBERNETES → server.close → nuqShutdown → shutdownWebhookQueue → shutdownIndexerQueue → shutdownPubSubLogging → shutdownTracing → process.exit(0); the harness also SIGTERMs child process groups and force-kills after 5s in dev.