firecrawl/firecrawl
github.com/firecrawl/firecrawl ↗ · 1858 files · 13.8 MB
--- 🔥 Firecrawl The API to search, scrape, and interact with the web at scale.
Overview
Firecrawl is a self-hostable web-scraping API monorepo: an Express 5/TypeScript service in apps/api exposes versioned REST endpoints (/v0, /v1, /v2) that turn URLs into markdown, JSON, screenshots and more, while background workers pull scrape jobs off a custom Postgres/FoundationDB-backed queue (NuQ) and run a waterfall of pluggable fetch engines (index cache, fire-engine Chrome CDP, Playwright microservice, plain fetch, PDF/document/image parsers). Billing (Autumn), auth (Supabase-backed API keys plus a keyless free tier), rate limiting (Redis), job state (Bigtable), and a web index cache (GCS + Postgres) are all first-class subsystems. Nine language SDKs, a Playwright sidecar, a Go html-to-markdown service, and a test-site app round out the monorepo.
System map
The core product is apps/api, an Express 5 server (src/index.ts) plus a family of worker processes spawned by a supervisor script (src/harness.ts). Jobs flow through NuQ, a bespoke queue layered on Postgres (apps/nuq-postgres/nuq.sql) with an experimental FoundationDB backend and RabbitMQ/PG-listen completion signaling; a few legacy BullMQ queues remain for LLMs.txt, deep-research and billing. Persistence is split across Redis (crawl state, rate limits, auth cache), Postgres via Drizzle (teams/API keys/Supabase), Bigtable (job state, index metadata), GCS (index documents, screenshots, logs) and ClickHouse (request logs). Scraping itself is delegated to a quality-ordered engine registry in scraper/scrapeURL/engines. Everything is configured through a single Zod schema in src/config.ts and deployed as Docker images via docker-compose.yaml and GitHub Actions.
- The HTTP entrypoint is apps/api/src/index.ts: it creates an express-ws app, mounts v0Router, /v1, /v2, /labs, /exchange and adminRouter, then registers a not-found handler and two error middlewares that map ZodError to 400 and everything else to a uuidv7-tagged 500.
- src/harness.ts is a process supervisor, not a server: it spawns the api process, legacy worker, NUQ_WORKER_COUNT nuq-workers plus prefetch/reconciler workers, extract and index workers, and manages Docker/Podman containers for nuq-postgres, RabbitMQ and FoundationDB when running locally.
- NuQ is a hand-rolled queue: class NuQ in services/worker/nuq.ts stores jobs in a Postgres table (nuq.queue_scrape, defined in apps/nuq-postgres/nuq.sql) and notifies waiters either through a dedicated PG LISTEN client or a RabbitMQ queue per worker.
- Four BullMQ queues still exist alongside NuQ — generate-llmstxt, deep-research, billing and precrawl — created lazily in services/queue-service.ts and surfaced through a bull-board UI mounted at /admin/<BULL_AUTH_KEY>/queues.
- Worker binaries are separate entrypoints under src/services: queue-worker.ts (BullMQ + NuQ scrape queue + monitor scheduler), worker/nuq-worker.ts, nuq-fdb-worker.ts, nuq-prefetch-worker.ts, nuq-reconciler-worker.ts, extract-worker.ts, indexing/index-worker.ts, zdr-worker.ts and cclog-worker.ts — each has its own package.json script.
- Scrape engines live in a single registry: Engine is a string-union type, engines[] is the enabled list gated by env config (PLAYWRIGHT_MICROSERVICE_URL, FIRE_PDF_BASE_URL, WIKIPEDIA_ENTERPRISE_*, XAI_API_KEY), and engineHandlers/engineOptions/engineMRTs map each engine to its implementation, feature support matrix and max-reasonable-time.
- HTML→markdown conversion is a Go shared library loaded through koffi FFI (lib/html-to-markdown.ts resolves sharedLibs/go-html-to-md via natives.ts), with an HTTP fallback (html-to-markdown-client.ts) to the standalone apps/go-html-to-md-service gorilla/mux service; @mendable/firecrawl-rs provides postProcessMarkdown.
- All configuration is a single Zod object in src/config.ts (638 lines) covering ports, DB URLs, feature envs like KEYLESS_CREDITS_PER_DAY, GOOGLE_WEB_RISK_API_KEY, NUQ_BACKEND and proxy credentials; modules import { config } directly.
- Drizzle ORM over node-postgres provides DB access: db/connection.ts builds named pools per a DB_POOL_PROFILE sizing scheme (db/pool-profiles.ts), returning null when a connection string is unset so features degrade gracefully.
- docker-compose.yaml defines the self-hosted deployment: api (built from apps/api, command node dist/src/harness.js --start-docker), playwright-service (apps/playwright-service-ts), redis, rabbitmq:3-management, nuq-postgres and an experimental foundationdb pair gated by NUQ_BACKEND=fdb.
- The monorepo ships nine SDK packages (js-sdk, python-sdk, go-sdk, rust-sdk, ruby-sdk, php-sdk, java-sdk, dot-net-sdk, elixir-sdk) each with its own manifest and publish workflow under .github/workflows/publish-*.yml.
- Observability is built in: OpenTelemetry tracing (src/otel.ts, lib/otel-tracer.ts withSpan/setSpanAttributes used across scrape paths), Prometheus metrics via prom-client (lib/http-metrics.ts, nuqGetLocalMetrics), and structured winston logging (lib/logger.ts).
Feature catalog
The v2 API (routes/v2.ts) is the primary surface: scrape, batch scrape, crawl (+status/errors/cancel/websocket), map, search (+developer/research proxies), extract/agent jobs, parse (multipart upload), browser/interact sessions, monitors, Slack integration, team usage/SIEM/threat-protection administration, and keyless free-tier endpoints. v1 mirrors the core endpoints (scrape/crawl/map/extract/search/deep-research/llms.txt) and v0 is a thin deprecated shim. Every mutating endpoint passes through the same middleware chain: authMiddleware (API key, OAuth, MCP delegated credential, or keyless), countryCheck, checkCreditsMiddleware (Autumn), blocklistMiddleware, and optionally idempotencyMiddleware.
- POST /v2/scrape runs scrapeController: zod-validates via scrapeRequestSchema, resolves safe mode and threat-protection policy, checks permissions and per-key format restrictions, acquires a team concurrency semaphore slot, then executes the scrape in-process via processJobInternal (skipNuq: true) rather than enqueueing.
- POST /v2/crawl creates a StoredCrawl in Redis (saveCrawl), registers a NuQ job group for concurrency/delay control, optionally expands a natural-language prompt into crawler options via generateCrawlerOptionsFromPrompt, then enqueues a mode:'kickoff' job that discovers links and fans out child scrape jobs.
- Crawl lifecycle endpoints: GET/DELETE /v2/crawl/:jobId (status via crawlStatusController, cancel via crawlCancelController), a WebSocket status channel on the same path, /crawl/ongoing|active, /crawl/:jobId/errors, /batch/scrape/:jobId reusing crawlStatusController, and /crawl/params-preview.
- POST /v2/map and /v2/search are credit-gated discovery endpoints; search chains providers fire_engine → searxng → DuckDuckGo (search/index.ts), and /v2/search/:jobId/feedback plus /v2/feedback collect quality signals.
- /v2/extract and /v2/agent are async job APIs (POST to start, GET /:jobId for status, agent also has trace/skill/snapshots/thread/cancel endpoints); both are billed at 20 credits via checkCreditsMiddleware(20) and extract carries a deprecationMiddleware('v2_extract') notice.
- Browser sessions are a first-class feature: POST/GET/DELETE /v2/browser (aliased /interact), /:sessionId/execute, /:sessionId/replay[/:pageId], and a public /browser/webhook/destroyed callback for the browser infra to report teardown.
- Monitors (scheduled re-scrapes with alerting) get full CRUD plus /monitor/:id/run, /checks history, and unauthenticated email confirm/unsubscribe endpoints; a scheduler in queue-worker.ts enqueues due checks via services/monitoring/scheduler.ts.
- A Slack app is embedded in the API: /v2/slack/oauth/start|callback, /status, /channels, DELETE /installation, plus unauthenticated /slack/commands and /slack/events verified by Slack's request signature (raw body captured by captureRawBody in index.ts).
- Team administration endpoints expose credit/token usage (current + historical), concurrency-check, queue-status, activity, threat-protection config incl. Zscaler test/sync/categories, and SIEM logging config with a test delivery route.
- /v2/parse accepts multipart file uploads (multer, 50MB cap, memoryStorage) or an uploadRef pointing at a prior PUT /v2/parse/upload/:uploadId body, converting documents to markdown via the same scrapeURL pipeline engines.
- A keyless free tier allows unauthenticated scrape/search/interact/parse requests capped per IPv4 per day when KEYLESS_CREDITS_PER_DAY and KEYLESS_REQUESTS_PER_DAY are both set; the trusted-proxy path uses x-firecrawl-keyless-ip authenticated by KEYLESS_PROXY_SECRET, with optional Spur VPN/TOR screening.
- v0 and v1 routers remain mounted: v0 exposes /v0/crawl, /v0/scrape, /v0/keyAuth and health endpoints behind deprecationMiddleware; v1 mirrors scrape/crawl/map/extract/search plus deep-research and generate-llmstxt status endpoints.
Data flow
Inbound JSON is validated by Zod schemas in controllers/v2/types.ts, then the request is stamped with team/org identity (ACUC), credits are checked against Autumn, and the job is either executed inline (sync scrape) or persisted as a NuQ row. The scrapeURL pipeline rewrites/permutes the URL, consults the blocklist, robots.txt, safe-mode and threat-protection policy, then walks a quality-ordered engine waterfall; each engine returns raw HTML which is normalized (removeUnwantedElements), converted to markdown (Go native lib or HTTP service), and passed through transformers (LLM extract, JSON, screenshots, change tracking) and postprocessors. Results are persisted to Bigtable job state, indexed to GCS+Postgres, billed per-feature, logged to PubSub/ClickHouse, and returned synchronously or delivered via webhook.
- Request validation is schema-first: scrapeRequestSchema.parse(req.body) runs inside an 'api.scrape.validate' OTel span, and ZodError is mapped centrally to a 400 with a strict 'Unrecognized key' message for v2.
- authenticateUser resolves the API key to an ACUC (auth credit usage chunk) via Redis-cached DB RPCs (authCreditUsageChunk), supports PREVIEW_TOKEN bypass, USE_DB_AUTHENTICATION mode, OAuth token introspection and MCP delegated credentials; the chunk carries team_id, org_id and feature flags used everywhere downstream.
- Credits are enforced by Autumn: checkCreditsMiddleware calls autumnService.checkCredits, fails open when Autumn is unreachable, clamps crawl/batch limits to remaining balance, and returns 402 with pricing links when denied; scrape-worker bills actual usage post-hoc via billScrapeJob → billTeam.
- Sync scrapes run a NuQJob in-process: scrapeController builds a NuQJob<ScrapeJobData> with skipNuq:true and calls processJobInternal directly inside teamConcurrencySemaphore.withSemaphore, so the queue row is skipped but the same worker code path executes.
- Async jobs persist as rows in nuq.queue_scrape via scrapeQueue.addJob/addJobs (queue-jobs.ts), with routing through nuq-router/scrapeQueueFdb when the FDB backend is selected; workers consume via runNuqWorker → processJobInternal.
- Inside scrapeURL, URL handling precedes engines: safe-mode resolution strips credentials/login actions, threat-protection checkUrl runs before any fetch with a per-scrape dedup map, robots.txt is consulted via shouldCheckRobots/fetchRobotsTxt, and urlSpecificParams/rewriteUrl normalize the target.
- buildFallbackList computes the engine waterfall: exchange/index short-circuits first, then engines are filtered by required FeatureFlags and sorted by quality (index=1000, wikipedia=500, fire-engine chrome-cdp=50, playwright=20, tlsclient=10, fetch=5, file engines=-20); scrapeURLLoopIter runs each and deems success on markdown length + status code + absence of page error.
- Engine output flows through htmlTransform (removeUnwantedElements), parseMarkdown, executeTransformers (llmExtract, deterministicJson, diff/changeTracking, audio/video, menu, product, query, sendToSearchIndex) and postprocessors (youtube) before becoming the Document returned to callers.
- Crawl state lives in Redis: StoredCrawl, per-URL locks (lockURL/lockURLsIndividually), job membership (addCrawlJob/addCrawlJobDone), robots.txt cache and crawl errors are managed by lib/crawl-redis.ts; completion fans out through crawlFinishedQueue to finishCrawlSuper.
- Terminal job state is written to Bigtable (job-state-store.ts, salted uuidv7 row keys) so scrape-status can read results after Redis TTLs expire; a SCRAPE_STATE_BARRIER_MS=2000 wait in scrape-worker lets the write land before sync responses return.
- The URL index is a two-tier cache: metadata/verdicts in Postgres via db/rpc.ts functions (queryIndexAtSplitLevel, queryEngpickerVerdict) plus document blobs in GCS (getIndexFromGCS), with a Redis index-cache layer and screenshot URL re-signing.
- Webhooks are queued and delivered asynchronously: createWebhookSender builds per-crawl senders, delivery.ts filters events and persists deliveries, and shutdownWebhookQueue is part of the graceful-exit chain in index.ts.
Behavior traces
Three flows were traced end-to-end. Startup: harness.ts supervises containers and child processes, then index.ts initializes blocklist/engine-forcing/exchange catalog and mounts routers. Sync scrape: middleware chain → scrapeController → semaphore → processJobInternal → startWebScraperPipeline → scrapeURL → engine waterfall → transforms → billing/logging → Document. Crawl: crawlController persists Redis crawl + NuQ group, enqueues a kickoff job, and workers discover links, enqueue child single_urls jobs, track completions via addCrawlJobDone and finish via crawl-logic.
- Startup (local dev): `pnpm harness` → tsx src/harness.ts --start → detectContainerRuntime() (docker|podman) → setupNuqPostgres() builds apps/nuq-postgres image, runs container, waits with pg SELECT 1 → setupNuqRabbitMQ() runs rabbitmq:3-management → spawns api (index.ts), worker, N nuq-workers, prefetch/reconciler, extract and index workers via execForward.
- API boot: index.ts loads dotenv+config, installs cacheableLookup on global http/https agents, registers body parsers with rawBody capture, mounts routers, then startServer() awaits initializeBlocklist(), initializeEngineForcing(), warmExchangeCatalog() before app.listen; SIGTERM drains via a 60s Kubernetes wait then nuqShutdown → webhook → indexer → pubsub → tracing shutdown.
- Scrape request, step order: requestTimingMiddleware → authMiddleware(Scrape, allowKeyless) → authenticateUser → countryCheck (geoip on req.ip when actions/headers/agent present) → checkCreditsMiddleware(1) → scrapeBlocklistMiddleware → scrapeController → scrapeRequestSchema.parse → resolveSafeMode → resolveThreatProtection → checkPermissions → checkKeyFormatRestriction → teamConcurrencySemaphore.withSemaphore → processJobInternal.
- Worker execution: processJobInternal wraps the job in zero-data-retention trace context → processJobWithTracing starts a Redis lock-extension interval for non-FDB jobs → dispatches on job.data.mode (kickoff → processKickoffJob, kickoff_sitemap → processKickoffSitemapJob, single_urls → processJob) → processJob races startWebScraperPipeline against remaining timeout → runWebScraper → scrapeURL.
- Engine waterfall: scrapeURL → buildMetaObject → threat checkUrl → buildFallbackList (exchange/index short-circuit, else feature-flag filtered quality sort) → scrapeURLLoop iterates engines calling scrapeURLWithEngine → scrapeURLLoopIter validates output (markdown non-empty, 2xx/304, no page error, auto-escalates stealthProxy on 401/403/429) → executeTransformers + postprocessors → Document.
- Crawl acceptance: crawlController → crawlRequestSchema.parse → safe-mode/threat/permission/key-restriction gates → optional prompt → buildPromptWithWebsiteStructure + generateCrawlerOptionsFromPrompt → merge with explicit options → initializeRequestCredits (Bigtable shards) → crawlToCrawler + getRobotsTxt → resolveNewGroupBackend → crawlGroup.addGroup → saveCrawl → markCrawlActive → _addScrapeJobToBullMQ(mode:'kickoff') → return {id, url}.
- Crawl fan-out (worker side): the kickoff job handler discovers links (WebCrawler/generateURLPermutations, sitemap via scrapeSitemap), checks discovered URLs against threat policy, then addCrawlJobs inserts child single_urls NuQ jobs; each child completion calls addCrawlJobDone and the last one pushes to crawlFinishedQueue handled by finishCrawlSuper.
- Persistence path for a scrape result: pipeline Document → GCS upload when saveScrapeResultToGCS → index write (saveIndexToGCS/upsertCachedIndexEntries) unless ZDR → writeScrapeJobState to Bigtable → billScrapeJob computes credits via calculateCreditsToBeBilled → logScrape → webhook sender emits crawl.page/scrape events → addCrawlJobDone when part of a crawl.
- Keyless request path: authMiddleware with allowKeyless → request lacks Bearer token → keyless branch resolves client IPv4 (or trusted x-firecrawl-keyless-ip) → optional Spur suspicious-IP check → consumeKeylessRequest decrements Redis daily counters → synthetic keylessTeamId(ip) → credits reserved via reserveKeylessCredits in the controller and refunded on failure via adjustKeylessCredits.
- Shutdown: SIGTERM → mcpActionLogRetention.stop() → 60s GCE drain wait under IS_KUBERNETES → server.close → nuqShutdown → shutdownWebhookQueue → shutdownIndexerQueue → shutdownPubSubLogging → shutdownTracing → process.exit(0); the harness also SIGTERMs child process groups and force-kills after 5s in dev.
Testing signals
Tests run under Vitest (vitest.config.ts, globals on, 120s timeouts, junit output) with ~112 test files under src/__tests__ plus colocated *.test.ts units. The dominant pattern is 'snips' — end-to-end tests hitting a live API brought up by `pnpm harness` — gated by env flags (TEST_SUITE_SELF_HOSTED, OPENAI_API_KEY, SEARXNG_ENDPOINT, PROXY_SERVER). CI (test-server.yml) runs a matrix over engine/proxy/search/ai/nuq-backend on self-hosted infra. Coverage is strong on the scrape pipeline and route surface; deep unit coverage exists for config-adjacent libs but many services/ workers have no dedicated tests.
- Test runner is Vitest with globals:true, environment node, include src/**/*.test.ts, testTimeout/hookTimeout 120s, junit reporter to test-results/junit.xml; a resolve alias redirects pdf-parse to skip its module.parent debug self-test.
- package.json test scripts split the suite by auth mode: test (exclude e2e_noAuth), test:local-no-auth (exclude e2e_withAuth), test:full, test:prod, and test:snips scoped to src/__tests__/snips/v1 and v2.
- Snips are true E2E: __tests__/snips/lib.ts derives TEST_API_URL/TEST_SUITE_WEBSITE from config, exposes describeIf/testIf gating helpers and capability flags (HAS_FIRE_ENGINE, HAS_PLAYWRIGHT, HAS_PROXY, HAS_AI), and refuses local TEST_SUITE_WEBSITE in production mode.
- The snips/v2 directory alone holds ~40 suites covering scrape, crawl, batch, map, parse, parsers/pdf variants, billing, concurrency, keyless, key-restriction, ip-restriction, idempotency (idmux), browser replay, monitors, agent auth/threads/effort, alexandria feedback and Express 5 behavior.
- AGENTS.md prescribes the workflow: write E2E 'snips' (1 happy + 1+ failure path), always use scrapeTimeout from ./lib, gate fire-engine tests with !TEST_SUITE_SELF_HOSTED and AI tests additionally on OPENAI_API_KEY/OLLAMA_BASE_URL, then run via `pnpm harness jest` (never pnpm start manually).
- CI test-server.yml runs a self-hosted matrix (playwright|fetch × proxy|no-proxy × searxng × openai × nuq postgres, plus one fdb-backend leg) with redis/rabbitmq services, installing FoundationDB client and setting NUQ_DATABASE_URL/NUQ_RABBITMQ_URL/USE_GO_MARKDOWN_PARSER/ALLOW_LOCAL_WEBHOOKS.
- Each SDK has a dedicated publish/test workflow (test-js-sdk, test-go-sdk, test-rust-sdk, test-dotnet-sdk, etc.) and repo-level workflows cover deploy-image staging/prod, nuq-postgres deploy, playwright deploy, redis deploy, scrape-evals and npm-audit remediation.
- Unit tests exist colocated for critical libs: exchange.test.ts, canonical-url.test.ts, crawl-redis.test.ts, job-state-store.test.ts, key-restriction.test.ts, image-ocr-gate.test.ts, ip-restriction.test.ts, oauth-token-introspection.test.ts, rate-limiter.test.ts, scrapeURL.test.ts and snips mocks under __tests__/snips/mocks.
- Coverage gaps: worker orchestration files (queue-worker.ts, nuq.ts, nuq-worker-runner.ts, harness.ts) and most services/monitoring, webhook delivery internals, and the go-html-to-md-service (only handler_test.go) have thin or no direct unit tests — behavior is mostly exercised through snips. (inferred)
- A dedicated test fixture app (apps/test-site, name firecrawl-test-site) and TEST_SUITE_WEBSITE=http://127.0.0.1:4321 in CI provide deterministic scrape targets; knip (knip.config.ts) plus a pre-commit hook enforces no unused exports/files.
Risk map
The highest blast radius sits in the auth/credit middleware chain and the scrapeURL pipeline: both are fail-open in places (Autumn outage, missing org_id) and carry many security gates that must run in the right order. Keyless free-tier abuse is mitigated by IPv4-only identity and Spur checks but is inherently risky. Shell-out exists in harness.ts (container commands are escaped) and via koffi FFI to a native lib. Secrets flow through env config; SIEM/webhook/ZDR paths handle sensitive customer data. Legacy BullMQ code, v0/v1 routers, and parallel PG/FDB queue backends add maintenance surface.
- checkCreditsMiddleware deliberately fails open when Autumn returns null and when req.auth.org_id is absent (keyless/preview/DB-auth bypass), so an Autumn outage or missing org mapping grants unlimited credits — explicitly commented as intentional to avoid customer outage.
- Keyless tier trusts a forwarded client IP only when x-firecrawl-keyless-secret matches KEYLESS_PROXY_SECRET; IPv6 clients are refused outright and Spur screening is optional (disabled without SPUR_API_KEY), so anonymized abuse controls depend on deployment config.
- harness.ts shells out to docker/podman with string commands; a shellEscape helper quotes credentials, but commands like `runtime ps --filter name=^${containerName}$` interpolate values — the file itself flags injection concerns in comments.
- Native FFI: lib/html-to-markdown.ts loads a .so/.dylib via koffi and calls ConvertHTMLToMarkdown on arbitrary scraped HTML; a malformed page or lib bug crashes the process, and GoMarkdownConverter.getInstance throws 'Go shared library not found' when sharedLibs is missing (e.g. partial Docker builds).
- The bull-board admin UI is gated only by knowledge of BULL_AUTH_KEY embedded in the URL path (/admin/<key>/queues); when unset the routes are disabled with a warning, but a leaked key exposes queue contents.
- Public unauthenticated endpoints exist by design: /v2/slack/commands + /slack/events (signature-verified), /browser/webhook/destroyed, /monitor/email/confirm + unsubscribe (token in body), /keyless/eligibility (secret-gated in controller), /mcp action-log ingest — each must verify its own credential since no authMiddleware runs.
- Zero-data-retention is a cross-cutting invariant: ZDR is resolved early in scrapeController specifically so the root OTel span never records the URL, and processJobInternal wraps execution in withZeroDataRetention; a missed code path anywhere leaks target URLs into traces/logs.
- Scrape pipeline has ~30 distinct error types (error.ts imports in scrapeURL/index.ts) mapped to ad-hoc status codes in the controller catch-block; new TransportableError codes must be added to the mapping or they fall through to generic 500s.
- Dual queue backends double the worker logic surface: scrape-worker branches on (job as any).backend === 'fdb' for lock extension, queue-jobs routes between scrapeQueue/scrapeQueueFdb/routedScrapeQueue, and nuq-postgres vs nuq-fdb diverge in prefetch/reconciler workers — bugs can affect one backend silently.
- Crawl children bypass request-time gates: discovered URLs rely on worker-side checks (isUrlBlocked post-scrape, checkUrlsAgainstThreatPolicy during link discovery, per-URL safe-mode in scrapeURL) so ordering bugs there would let a seed-approved crawl scrape disallowed pages.
- Supabase/Postgres pools are sized against a shared pgbouncer cap ('pods * max' per pool per comments in db/connection.ts); a misconfigured DB_POOL_PROFILE or missing keepWarm behavior can exhaust the 12000-connection pooler budget fleet-wide.
- Deprecated surfaces remain live: v0 router, /v2/extract (deprecationMiddleware), /v2/research and /developer legacy proxies, plus a v1 API surface — each is a maintenance and audit surface that still executes production code paths.
Extension points
The cleanest extension seams are the scrape engine registry (add an Engine string, handler, MRT and feature/quality entry), the transformer/postprocessor lists, and the route tables (v2Router wiring controllers through shared middlewares). NuQ job modes ('single_urls', 'kickoff', 'kickoff_sitemap') dispatch in processJobWithTracing, search providers chain in search/index.ts, threat-protection providers plug behind lib/threat-protection, and exchange eligibility is isolated in lib/exchange.ts. New env config goes in the Zod schema; new team flags ride TeamFlags in the ACUC.
- Adding a scrape engine = extend the Engine union, append to engines[] behind an env gate, and fill engineHandlers, engineMRTs and engineOptions (feature support + quality) — buildFallbackList and the waterfall pick it up automatically.
- Feature flags are declarative: the featureFlags tuple plus featureFlagOptions priorities drive which engines qualify; adding a flag requires updating the union, every engine's features map, and the format/options schema in controllers/v2/types.ts.
- New output formats plug in as transformers: executeTransformers runs modules like llmExtract, deterministicJson, diff (changeTracking), menu, product, audio, video, sendToSearchIndex; postprocessors (youtube) run after.
- Routes are registered flat in routes/v2.ts with per-route middleware composition; a new endpoint needs a controller under controllers/v2, a RateLimiterMode in types.ts, and the standard auth/credits/blocklist middleware stack.
- NuQ job dispatch is a mode switch: ScrapeJobData.mode ('single_urls'|'kickoff'|'kickoff_sitemap') selects processJob/processKickoffJob/processKickoffSitemapJob in processJobWithTracing — new job kinds extend the union in types.ts and add a branch.
- Search providers chain by config: search/index.ts tries fire_engine_search when FIRE_ENGINE_BETA_URL is set, then searxng_search when SEARXNG_ENDPOINT is set, then ddgSearch — a new provider slots into that fallback chain.
- URL-specific behavior is table-driven: urlSpecificParams overrides options per domain, engine-forcing (getEngineForUrl/initializeEngineForcing) pins engines per host, and the blocklist (initializeBlocklist/isUrlBlocked) is a warmed data file — all three are data-shaped extension points.
- Threat protection is provider-pluggable: lib/threat-protection resolves a policy per team/request and checkUrl consults providers (Google Web Risk normal/ZDR-synced modes, Zscaler ZIA), with org config exposed via /v2/team/threat-protection routes.
- Billing features are keyed by Autumn feature IDs (CREDITS_FEATURE_ID, SEARCH_CREDITS_FEATURE_ID, featureIdForBillingEndpoint) — new metered endpoints pass their featureId through checkCreditsMiddleware's second arg.
- SDKs are independently versioned packages under apps/*-sdk with per-language publish workflows; the API surface they consume is documented by openapi.json/v1-openapi.json/openapi-v0.json in apps/api.
- Config extension is centralized: new env vars require a Zod entry in configSchema (with helpers like emptyStringAsUndefined/delimitedList) and are then read as config.X everywhere; MCP action-log config even validates at startup via getMcpActionLogConfigErrors.
Contributor handoff
Work happens almost entirely in apps/api. Read AGENTS.md/CLAUDE.md first — they mandate E2E 'snips' over unit tests, `pnpm harness jest` (never pnpm start), and fixing knip failures rather than bypassing hooks. Safe edit boundaries: a new engine touches only engines/* plus the registry; a new endpoint touches controllers/v2 + routes/v2.ts + a RateLimiterMode; a new transformer touches transformers/. Danger zones: the auth/credit middleware order, ZDR propagation, and anything changing job payload shape (NuQ rows in flight). Always add env config to the Zod schema and gate tests by capability flags.
- AGENTS.md is the contract: write snips (1 happy + 1+ failure paths), use scrapeTimeout from ./lib for scrape timeouts, gate fire-engine tests on !TEST_SUITE_SELF_HOSTED and AI tests on OPENAI_API_KEY/OLLAMA_BASE_URL, run via `pnpm harness jest`, push a PR and let CI run the full suite.
- CLAUDE.md adds: never bypass the pre-commit knip check (fix unused exports/files even if preexisting); package.json wires husky+lint-staged+knip so commits enforce prettier formatting and dead-code hygiene.
- Read first for scrape work: controllers/v2/types.ts (2825-line zod schema for all request/response types), scraper/scrapeURL/index.ts (1926-line pipeline), engines/index.ts (registry), and lib/error.ts + scrapeURL/error.ts for the TransportableError taxonomy.
- Middleware order is load-bearing: requestTiming → authMiddleware → countryCheck → checkCredits → blocklist → controller; reordering or omitting one (e.g. on a new route) silently drops auth, geo-restriction, billing or blocklist enforcement.
- Zero-data-retention must be threaded: ZDR suppresses OTel recording, GCS/index writes, and URL logging; any new code path that touches a scrape must check internalOptions.zeroDataRetention / getScrapeZDR before persisting or tracing.
- Tests require a running stack: `pnpm harness` spawns api+workers+containers; snips hit TEST_API_URL with TEST_API_KEY='test'/TEST_TEAM_ID='bypass' in self-host mode — do not hand-start the server, and scope test runs to the relevant suite since the full suite is slow.
- Naming traps: _addScrapeJobToBullMQ actually enqueues onto NuQ (BullMQ naming is vestigial); 'snips' means E2E tests; 'ACUC' is the auth/credit chunk; 'NuQ' queues live in Postgres tables prefixed nuq.; job 'mode' (not endpoint) selects worker behavior.
- Error contract: controllers return {success:false, error} with status codes mapped per TransportableError code in scrape.ts's catch block — new error codes need an explicit branch (e.g. SCRAPE_DNS_RESOLUTION_ERROR→200, SCRAPE_NO_CACHED_DATA→404) or they surface as UNKNOWN_ERROR 500s.
- Env/config changes: add the var to the Zod schema in config.ts (empty-string handling matters — use emptyStringAsUndefined), reference it via config.X, and update docker-compose.yaml common-env + CI workflow envs if the feature must work self-hosted.
- In-flight job compatibility: NuQ rows are JSONB payloads consumed by separate worker binaries, so changing ScrapeJobData/internalOptions shape requires backward-compat handling (the code already guards with ?? fallbacks like job.data.internalOptions?.orgId ?? null).
Languages
| Language | Files | Share of source |
|---|---|---|
| TypeScript | 955 | 65% |
| Python | 182 | 12% |
| Markdown | 100 | docs/config |
| Java | 93 | 6% |
| YAML | 76 | docs/config |
| PHP | 74 | 5% |
| C# | 41 | 3% |
| JSON | 37 | docs/config |
Start reading here
Probable source entrypoint: apps/ — inferred from the directory tree and manifest, not the upstream docs.
Source roots
apps/ (1616 files) skills/ (16 files) firecrawl-cli-skills/ (1 files)
Source layout — apps/
| Path | Files |
|---|---|
apps/ruby-sdk/ | 32 |
apps/ui/ | 30 |
Repository structure
| Top-level path | Files |
|---|---|
apps/ | 1616 |
examples/ | 167 |
.github/ | 45 |
skills/ | 16 |
firecrawl-cli-skills/ | 1 |
firecrawl-workflows/ | 1 |
firecrawl-cli/ | 1 |
firecrawl-skills/ | 1 |
Docs found: AGENTS.mdCLAUDE.mdCONTRIBUTING.mdREADME.md
Package
apps/api/package.json — firecrawl-scraper-js v1.0.0, 80 declared dependencies.
@ai-sdk/anthropic@ai-sdk/deepinfra@ai-sdk/fireworks@ai-sdk/google@ai-sdk/google-vertex@ai-sdk/groq@ai-sdk/openai@ai-sdk/xai@bull-board/api@bull-board/express@clickhouse/client@dqbd/tiktoken
CI & config entrypoints
README.md— Repository readmeAGENTS.md— Agent instructionsCLAUDE.md— Claude instructions.github/workflows/deploy-image-staging.yml— GitHub Actions workflow.github/workflows/publish-python-sdk.yml— GitHub Actions workflow.github/workflows/validate-lockfiles.yml— GitHub Actions workflow.github/workflows/sync-skills-catalog.yml— GitHub Actions workflow.github/workflows/publish-js-sdk.yml— GitHub Actions workflow.github/workflows/test-dotnet-sdk.yml— GitHub Actions workflow.github/workflows/deploy-redis.yml— GitHub Actions workflow
Structural health signals
Average score 6/10 across 96 analyzed source files; 67 hotspots surfaced.
apps/js-sdk/firecrawl/src/index.backup.ts— 2146 lines, score 4 (Large file, Deep nesting)apps/api/src/services/worker/nuq.ts— 1675 lines, score 4 (Large file, Deep nesting)apps/api/src/services/indexing/index-worker.ts— 810 lines, score 4.5 (Large file, Deep nesting)
These are deterministic review leads — size, churn, and structural signals — not proof of a bug in this project.
Recent history
556c12c9d2932026-09-21 — Nsc/fix rate limits alexandria (#4725)04034c3fc9c82026-09-21 — feat(api): store Alexandria feedback in dedicated tables (#4722)024261beb9762026-09-21 — Nick: (#4721)a6e2130d72b72026-09-21 — Use Alexandria name in provider execution errors (#4720)504905318fe82026-09-21 — feat(api): refuse free-plan teams on paid-plan-only Exchange capabilities (#4716)