# Behavior traces — firecrawl/firecrawl

> Canonical page: https://codevetter.com/unpack/firecrawl/behavior-traces

# Behavior traces — firecrawl/firecrawl

Scanned commit 556c12c9d293bc7392feb008946944d54068d04c (2026-09-21). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

Three flows were traced end-to-end. Startup: harness.ts supervises containers and child processes, then index.ts initializes blocklist/engine-forcing/exchange catalog and mounts routers. Sync scrape: middleware chain → scrapeController → semaphore → processJobInternal → startWebScraperPipeline → scrapeURL → engine waterfall → transforms → billing/logging → Document. Crawl: crawlController persists Redis crawl + NuQ group, enqueues a kickoff job, and workers discover links, enqueue child single_urls jobs, track completions via addCrawlJobDone and finish via crawl-logic.
- Startup (local dev): `pnpm harness` → tsx src/harness.ts --start → detectContainerRuntime() (docker|podman) → setupNuqPostgres() builds apps/nuq-postgres image, runs container, waits with pg SELECT 1 → setupNuqRabbitMQ() runs rabbitmq:3-management → spawns api (index.ts), worker, N nuq-workers, prefetch/reconciler, extract and index workers via execForward. — [apps/api/src/harness.ts#L397-409](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/harness.ts#L397-L409) [apps/api/src/harness.ts#L516-584](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/harness.ts#L516-L584) [apps/api/package.json#L22-30](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/package.json#L22-L30)
- API boot: index.ts loads dotenv+config, installs cacheableLookup on global http/https agents, registers body parsers with rawBody capture, mounts routers, then startServer() awaits initializeBlocklist(), initializeEngineForcing(), warmExchangeCatalog() before app.listen; SIGTERM drains via a 60s Kubernetes wait then nuqShutdown → webhook → indexer → pubsub → tracing shutdown. — [apps/api/src/index.ts#L55-68](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L55-L68) [apps/api/src/index.ts#L143-196](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L143-L196)
- Scrape request, step order: requestTimingMiddleware → authMiddleware(Scrape, allowKeyless) → authenticateUser → countryCheck (geoip on req.ip when actions/headers/agent present) → checkCreditsMiddleware(1) → scrapeBlocklistMiddleware → scrapeController → scrapeRequestSchema.parse → resolveSafeMode → resolveThreatProtection → checkPermissions → checkKeyFormatRestriction → teamConcurrencySemaphore.withSemaphore → processJobInternal. — [apps/api/src/routes/v2.ts#L235-242](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/routes/v2.ts#L235-L242) [apps/api/src/routes/shared.ts#L271-478](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/routes/shared.ts#L271-L478) [apps/api/src/controllers/v2/scrape.ts#L57-210](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/controllers/v2/scrape.ts#L57-L210)
- Worker execution: processJobInternal wraps the job in zero-data-retention trace context → processJobWithTracing starts a Redis lock-extension interval for non-FDB jobs → dispatches on job.data.mode (kickoff → processKickoffJob, kickoff_sitemap → processKickoffSitemapJob, single_urls → processJob) → processJob races startWebScraperPipeline against remaining timeout → runWebScraper → scrapeURL. — [apps/api/src/services/worker/scrape-worker.ts#L1765-1840](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/scrape-worker.ts#L1765-L1840) [apps/api/src/services/worker/scrape-worker.ts#L392-465](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/scrape-worker.ts#L392-L465) [apps/api/src/main/runWebScraper.ts#L9-97](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/main/runWebScraper.ts#L9-L97)
- Engine waterfall: scrapeURL → buildMetaObject → threat checkUrl → buildFallbackList (exchange/index short-circuit, else feature-flag filtered quality sort) → scrapeURLLoop iterates engines calling scrapeURLWithEngine → scrapeURLLoopIter validates output (markdown non-empty, 2xx/304, no page error, auto-escalates stealthProxy on 401/403/429) → executeTransformers + postprocessors → Document. — [apps/api/src/scraper/scrapeURL/index.ts#L821-870](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/index.ts#L821-L870) [apps/api/src/scraper/scrapeURL/index.ts#L662-807](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/index.ts#L662-L807) [apps/api/src/scraper/scrapeURL/engines/index.ts#L637-734](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/engines/index.ts#L637-L734)
- Crawl acceptance: crawlController → crawlRequestSchema.parse → safe-mode/threat/permission/key-restriction gates → optional prompt → buildPromptWithWebsiteStructure + generateCrawlerOptionsFromPrompt → merge with explicit options → initializeRequestCredits (Bigtable shards) → crawlToCrawler + getRobotsTxt → resolveNewGroupBackend → crawlGroup.addGroup → saveCrawl → markCrawlActive → _addScrapeJobToBullMQ(mode:'kickoff') → return {id, url}. — [apps/api/src/controllers/v2/crawl.ts#L49-177](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/controllers/v2/crawl.ts#L49-L177) [apps/api/src/controllers/v2/crawl.ts#L301-412](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/controllers/v2/crawl.ts#L301-L412)
- Crawl fan-out (worker side): the kickoff job handler discovers links (WebCrawler/generateURLPermutations, sitemap via scrapeSitemap), checks discovered URLs against threat policy, then addCrawlJobs inserts child single_urls NuQ jobs; each child completion calls addCrawlJobDone and the last one pushes to crawlFinishedQueue handled by finishCrawlSuper. — [apps/api/src/services/worker/scrape-worker.ts#L14-31](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/scrape-worker.ts#L14-L31) [apps/api/src/services/worker/scrape-worker.ts#L1557-1724](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/scrape-worker.ts#L1557-L1724) [apps/api/src/services/worker/crawl-logic.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/crawl-logic.ts) [apps/api/src/scraper/crawler/sitemap.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/crawler/sitemap.ts)
- Persistence path for a scrape result: pipeline Document → GCS upload when saveScrapeResultToGCS → index write (saveIndexToGCS/upsertCachedIndexEntries) unless ZDR → writeScrapeJobState to Bigtable → billScrapeJob computes credits via calculateCreditsToBeBilled → logScrape → webhook sender emits crawl.page/scrape events → addCrawlJobDone when part of a crawl. — [apps/api/src/services/worker/scrape-worker.ts#L819-1021](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/scrape-worker.ts#L819-L1021) [apps/api/src/services/index.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/index.ts) [apps/api/src/lib/gcs-jobs.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/lib/gcs-jobs.ts)
- Keyless request path: authMiddleware with allowKeyless → request lacks Bearer token → keyless branch resolves client IPv4 (or trusted x-firecrawl-keyless-ip) → optional Spur suspicious-IP check → consumeKeylessRequest decrements Redis daily counters → synthetic keylessTeamId(ip) → credits reserved via reserveKeylessCredits in the controller and refunded on failure via adjustKeylessCredits. — [apps/api/src/controllers/auth.ts#L466-630](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/controllers/auth.ts#L466-L630) [apps/api/src/controllers/v2/scrape.ts#L247-270](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/controllers/v2/scrape.ts#L247-L270) [apps/api/src/lib/keyless.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/lib/keyless.ts)
- Shutdown: SIGTERM → mcpActionLogRetention.stop() → 60s GCE drain wait under IS_KUBERNETES → server.close → nuqShutdown → shutdownWebhookQueue → shutdownIndexerQueue → shutdownPubSubLogging → shutdownTracing → process.exit(0); the harness also SIGTERMs child process groups and force-kills after 5s in dev. — [apps/api/src/index.ts#L168-194](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L168-L194) [apps/api/src/harness.ts#L329-395](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/harness.ts#L329-L395)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)