CodeVetter · Repo Unpack · Risk map
firecrawl/firecrawl
The highest blast radius sits in the auth/credit middleware chain and the scrapeURL pipeline: both are fail-open in places (Autumn outage, missing org_id) and carry many security gates that must run in the right order. Keyless free-tier abuse is mitigated by IPv4-only identity and Spur checks but is inherently risky. Shell-out exists in harness.ts (container commands are escaped) and via koffi FFI to a native lib. Secrets flow through env config; SIEM/webhook/ZDR paths handle sensitive customer data. Legacy BullMQ code, v0/v1 routers, and parallel PG/FDB queue backends add maintenance surface.
- checkCreditsMiddleware deliberately fails open when Autumn returns null and when req.auth.org_id is absent (keyless/preview/DB-auth bypass), so an Autumn outage or missing org mapping grants unlimited credits — explicitly commented as intentional to avoid customer outage.
- Keyless tier trusts a forwarded client IP only when x-firecrawl-keyless-secret matches KEYLESS_PROXY_SECRET; IPv6 clients are refused outright and Spur screening is optional (disabled without SPUR_API_KEY), so anonymized abuse controls depend on deployment config.
- harness.ts shells out to docker/podman with string commands; a shellEscape helper quotes credentials, but commands like `runtime ps --filter name=^${containerName}$` interpolate values — the file itself flags injection concerns in comments.
- Native FFI: lib/html-to-markdown.ts loads a .so/.dylib via koffi and calls ConvertHTMLToMarkdown on arbitrary scraped HTML; a malformed page or lib bug crashes the process, and GoMarkdownConverter.getInstance throws 'Go shared library not found' when sharedLibs is missing (e.g. partial Docker builds).
- The bull-board admin UI is gated only by knowledge of BULL_AUTH_KEY embedded in the URL path (/admin/<key>/queues); when unset the routes are disabled with a warning, but a leaked key exposes queue contents.
- Public unauthenticated endpoints exist by design: /v2/slack/commands + /slack/events (signature-verified), /browser/webhook/destroyed, /monitor/email/confirm + unsubscribe (token in body), /keyless/eligibility (secret-gated in controller), /mcp action-log ingest — each must verify its own credential since no authMiddleware runs.
- Zero-data-retention is a cross-cutting invariant: ZDR is resolved early in scrapeController specifically so the root OTel span never records the URL, and processJobInternal wraps execution in withZeroDataRetention; a missed code path anywhere leaks target URLs into traces/logs.
- Scrape pipeline has ~30 distinct error types (error.ts imports in scrapeURL/index.ts) mapped to ad-hoc status codes in the controller catch-block; new TransportableError codes must be added to the mapping or they fall through to generic 500s.
- Dual queue backends double the worker logic surface: scrape-worker branches on (job as any).backend === 'fdb' for lock extension, queue-jobs routes between scrapeQueue/scrapeQueueFdb/routedScrapeQueue, and nuq-postgres vs nuq-fdb diverge in prefetch/reconciler workers — bugs can affect one backend silently.
- Crawl children bypass request-time gates: discovered URLs rely on worker-side checks (isUrlBlocked post-scrape, checkUrlsAgainstThreatPolicy during link discovery, per-URL safe-mode in scrapeURL) so ordering bugs there would let a seed-approved crawl scrape disallowed pages.
- Supabase/Postgres pools are sized against a shared pgbouncer cap ('pods * max' per pool per comments in db/connection.ts); a misconfigured DB_POOL_PROFILE or missing keepWarm behavior can exhaust the 12000-connection pooler budget fleet-wide.
- Deprecated surfaces remain live: v0 router, /v2/extract (deprecationMiddleware), /v2/research and /developer legacy proxies, plus a v1 API surface — each is a maintenance and audit surface that still executes production code paths.