firecrawl/firecrawl
The core product is apps/api, an Express 5 server (src/index.ts) plus a family of worker processes spawned by a supervisor script (src/harness.ts). Jobs flow through NuQ, a bespoke queue layered on Postgres (apps/nuq-postgres/nuq.sql) with an experimental FoundationDB backend and RabbitMQ/PG-listen completion signaling; a few legacy BullMQ queues remain for LLMs.txt, deep-research and billing. Persistence is split across Redis (crawl state, rate limits, auth cache), Postgres via Drizzle (teams/API keys/Supabase), Bigtable (job state, index metadata), GCS (index documents, screenshots, logs) and ClickHouse (request logs). Scraping itself is delegated to a quality-ordered engine registry in scraper/scrapeURL/engines. Everything is configured through a single Zod schema in src/config.ts and deployed as Docker images via docker-compose.yaml and GitHub Actions.
- The HTTP entrypoint is apps/api/src/index.ts: it creates an express-ws app, mounts v0Router, /v1, /v2, /labs, /exchange and adminRouter, then registers a not-found handler and two error middlewares that map ZodError to 400 and everything else to a uuidv7-tagged 500.
- src/harness.ts is a process supervisor, not a server: it spawns the api process, legacy worker, NUQ_WORKER_COUNT nuq-workers plus prefetch/reconciler workers, extract and index workers, and manages Docker/Podman containers for nuq-postgres, RabbitMQ and FoundationDB when running locally.
- NuQ is a hand-rolled queue: class NuQ in services/worker/nuq.ts stores jobs in a Postgres table (nuq.queue_scrape, defined in apps/nuq-postgres/nuq.sql) and notifies waiters either through a dedicated PG LISTEN client or a RabbitMQ queue per worker.
- Four BullMQ queues still exist alongside NuQ — generate-llmstxt, deep-research, billing and precrawl — created lazily in services/queue-service.ts and surfaced through a bull-board UI mounted at /admin/<BULL_AUTH_KEY>/queues.
- Worker binaries are separate entrypoints under src/services: queue-worker.ts (BullMQ + NuQ scrape queue + monitor scheduler), worker/nuq-worker.ts, nuq-fdb-worker.ts, nuq-prefetch-worker.ts, nuq-reconciler-worker.ts, extract-worker.ts, indexing/index-worker.ts, zdr-worker.ts and cclog-worker.ts — each has its own package.json script.
- Scrape engines live in a single registry: Engine is a string-union type, engines[] is the enabled list gated by env config (PLAYWRIGHT_MICROSERVICE_URL, FIRE_PDF_BASE_URL, WIKIPEDIA_ENTERPRISE_*, XAI_API_KEY), and engineHandlers/engineOptions/engineMRTs map each engine to its implementation, feature support matrix and max-reasonable-time.
- HTML→markdown conversion is a Go shared library loaded through koffi FFI (lib/html-to-markdown.ts resolves sharedLibs/go-html-to-md via natives.ts), with an HTTP fallback (html-to-markdown-client.ts) to the standalone apps/go-html-to-md-service gorilla/mux service; @mendable/firecrawl-rs provides postProcessMarkdown.
- All configuration is a single Zod object in src/config.ts (638 lines) covering ports, DB URLs, feature envs like KEYLESS_CREDITS_PER_DAY, GOOGLE_WEB_RISK_API_KEY, NUQ_BACKEND and proxy credentials; modules import { config } directly.
- Drizzle ORM over node-postgres provides DB access: db/connection.ts builds named pools per a DB_POOL_PROFILE sizing scheme (db/pool-profiles.ts), returning null when a connection string is unset so features degrade gracefully.
- docker-compose.yaml defines the self-hosted deployment: api (built from apps/api, command node dist/src/harness.js --start-docker), playwright-service (apps/playwright-service-ts), redis, rabbitmq:3-management, nuq-postgres and an experimental foundationdb pair gated by NUQ_BACKEND=fdb.
- The monorepo ships nine SDK packages (js-sdk, python-sdk, go-sdk, rust-sdk, ruby-sdk, php-sdk, java-sdk, dot-net-sdk, elixir-sdk) each with its own manifest and publish workflow under .github/workflows/publish-*.yml.
- Observability is built in: OpenTelemetry tracing (src/otel.ts, lib/otel-tracer.ts withSpan/setSpanAttributes used across scrape paths), Prometheus metrics via prom-client (lib/http-metrics.ts, nuqGetLocalMetrics), and structured winston logging (lib/logger.ts).