# System map — firecrawl/firecrawl

> Canonical page: https://codevetter.com/unpack/firecrawl/system-map

# System map — firecrawl/firecrawl

Scanned commit 556c12c9d293bc7392feb008946944d54068d04c (2026-09-21). Generated by codevetter codevetter 1.12.0 on 2026-09-22.

The core product is apps/api, an Express 5 server (src/index.ts) plus a family of worker processes spawned by a supervisor script (src/harness.ts). Jobs flow through NuQ, a bespoke queue layered on Postgres (apps/nuq-postgres/nuq.sql) with an experimental FoundationDB backend and RabbitMQ/PG-listen completion signaling; a few legacy BullMQ queues remain for LLMs.txt, deep-research and billing. Persistence is split across Redis (crawl state, rate limits, auth cache), Postgres via Drizzle (teams/API keys/Supabase), Bigtable (job state, index metadata), GCS (index documents, screenshots, logs) and ClickHouse (request logs). Scraping itself is delegated to a quality-ordered engine registry in scraper/scrapeURL/engines. Everything is configured through a single Zod schema in src/config.ts and deployed as Docker images via docker-compose.yaml and GitHub Actions.
- The HTTP entrypoint is apps/api/src/index.ts: it creates an express-ws app, mounts v0Router, /v1, /v2, /labs, /exchange and adminRouter, then registers a not-found handler and two error middlewares that map ZodError to 400 and everything else to a uuidv7-tagged 500. — [apps/api/src/index.ts#L121-138](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L121-L138) [apps/api/src/index.ts#L209-310](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L209-L310)
- src/harness.ts is a process supervisor, not a server: it spawns the api process, legacy worker, NUQ_WORKER_COUNT nuq-workers plus prefetch/reconciler workers, extract and index workers, and manages Docker/Podman containers for nuq-postgres, RabbitMQ and FoundationDB when running locally. — [apps/api/src/harness.ts#L43-64](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/harness.ts#L43-L64) [apps/api/src/harness.ts#L516-670](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/harness.ts#L516-L670)
- NuQ is a hand-rolled queue: class NuQ in services/worker/nuq.ts stores jobs in a Postgres table (nuq.queue_scrape, defined in apps/nuq-postgres/nuq.sql) and notifies waiters either through a dedicated PG LISTEN client or a RabbitMQ queue per worker. — [apps/api/src/services/worker/nuq.ts#L94-120](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/nuq.ts#L94-L120) [apps/api/src/services/worker/nuq.ts#L1755-1764](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/nuq.ts#L1755-L1764) [apps/nuq-postgres/nuq.sql#L44-60](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/nuq-postgres/nuq.sql#L44-L60)
- Four BullMQ queues still exist alongside NuQ — generate-llmstxt, deep-research, billing and precrawl — created lazily in services/queue-service.ts and surfaced through a bull-board UI mounted at /admin/<BULL_AUTH_KEY>/queues. — [apps/api/src/services/queue-service.ts#L41-97](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/queue-service.ts#L41-L97) [apps/api/src/index.ts#L100-119](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/index.ts#L100-L119)
- Worker binaries are separate entrypoints under src/services: queue-worker.ts (BullMQ + NuQ scrape queue + monitor scheduler), worker/nuq-worker.ts, nuq-fdb-worker.ts, nuq-prefetch-worker.ts, nuq-reconciler-worker.ts, extract-worker.ts, indexing/index-worker.ts, zdr-worker.ts and cclog-worker.ts — each has its own package.json script. — [apps/api/package.json#L23-40](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/package.json#L23-L40) [apps/api/src/services/worker/nuq-worker.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/worker/nuq-worker.ts) [apps/api/src/services/queue-worker.ts#L1-37](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/services/queue-worker.ts#L1-L37)
- Scrape engines live in a single registry: Engine is a string-union type, engines[] is the enabled list gated by env config (PLAYWRIGHT_MICROSERVICE_URL, FIRE_PDF_BASE_URL, WIKIPEDIA_ENTERPRISE_*, XAI_API_KEY), and engineHandlers/engineOptions/engineMRTs map each engine to its implementation, feature support matrix and max-reasonable-time. — [apps/api/src/scraper/scrapeURL/engines/index.ts#L51-103](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/engines/index.ts#L51-L103) [apps/api/src/scraper/scrapeURL/engines/index.ts#L193-239](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/engines/index.ts#L193-L239)
- HTML→markdown conversion is a Go shared library loaded through koffi FFI (lib/html-to-markdown.ts resolves sharedLibs/go-html-to-md via natives.ts), with an HTTP fallback (html-to-markdown-client.ts) to the standalone apps/go-html-to-md-service gorilla/mux service; @mendable/firecrawl-rs provides postProcessMarkdown. — [apps/api/src/lib/html-to-markdown.ts#L1-45](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/lib/html-to-markdown.ts#L1-L45) [apps/api/src/natives.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/natives.ts) [apps/go-html-to-md-service/main.go#L1-40](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/go-html-to-md-service/main.go#L1-L40)
- All configuration is a single Zod object in src/config.ts (638 lines) covering ports, DB URLs, feature envs like KEYLESS_CREDITS_PER_DAY, GOOGLE_WEB_RISK_API_KEY, NUQ_BACKEND and proxy credentials; modules import { config } directly. — [apps/api/src/config.ts#L71-150](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/config.ts#L71-L150)
- Drizzle ORM over node-postgres provides DB access: db/connection.ts builds named pools per a DB_POOL_PROFILE sizing scheme (db/pool-profiles.ts), returning null when a connection string is unset so features degrade gracefully. — [apps/api/src/db/connection.ts#L1-50](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/db/connection.ts#L1-L50)
- docker-compose.yaml defines the self-hosted deployment: api (built from apps/api, command node dist/src/harness.js --start-docker), playwright-service (apps/playwright-service-ts), redis, rabbitmq:3-management, nuq-postgres and an experimental foundationdb pair gated by NUQ_BACKEND=fdb. — [docker-compose.yaml#L62-128](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/docker-compose.yaml#L62-L128) [docker-compose.yaml#L147-227](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/docker-compose.yaml#L147-L227)
- The monorepo ships nine SDK packages (js-sdk, python-sdk, go-sdk, rust-sdk, ruby-sdk, php-sdk, java-sdk, dot-net-sdk, elixir-sdk) each with its own manifest and publish workflow under .github/workflows/publish-*.yml. — [apps/api/package.json](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/package.json) [.github/workflows/publish-js-sdk.yml](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/.github/workflows/publish-js-sdk.yml) [docker-compose.yaml](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/docker-compose.yaml)
- Observability is built in: OpenTelemetry tracing (src/otel.ts, lib/otel-tracer.ts withSpan/setSpanAttributes used across scrape paths), Prometheus metrics via prom-client (lib/http-metrics.ts, nuqGetLocalMetrics), and structured winston logging (lib/logger.ts). — [apps/api/src/otel.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/otel.ts) [apps/api/src/lib/otel-tracer.ts](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/lib/otel-tracer.ts) [apps/api/src/scraper/scrapeURL/index.ts#L3-7](https://github.com/firecrawl/firecrawl/blob/556c12c9d293bc7392feb008946944d54068d04c/apps/api/src/scraper/scrapeURL/index.ts#L3-L7)

Every claim links to its cited files at the pinned commit. Citations identify source evidence; they do not establish runtime correctness. Independent CodeVetter scan — not reviewed by upstream maintainers.

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)