CodeVetter · Repo Unpack · Contributor handoff
firecrawl/firecrawl
Work happens almost entirely in apps/api. Read AGENTS.md/CLAUDE.md first — they mandate E2E 'snips' over unit tests, `pnpm harness jest` (never pnpm start), and fixing knip failures rather than bypassing hooks. Safe edit boundaries: a new engine touches only engines/* plus the registry; a new endpoint touches controllers/v2 + routes/v2.ts + a RateLimiterMode; a new transformer touches transformers/. Danger zones: the auth/credit middleware order, ZDR propagation, and anything changing job payload shape (NuQ rows in flight). Always add env config to the Zod schema and gate tests by capability flags.
- AGENTS.md is the contract: write snips (1 happy + 1+ failure paths), use scrapeTimeout from ./lib for scrape timeouts, gate fire-engine tests on !TEST_SUITE_SELF_HOSTED and AI tests on OPENAI_API_KEY/OLLAMA_BASE_URL, run via `pnpm harness jest`, push a PR and let CI run the full suite.
- CLAUDE.md adds: never bypass the pre-commit knip check (fix unused exports/files even if preexisting); package.json wires husky+lint-staged+knip so commits enforce prettier formatting and dead-code hygiene.
- Read first for scrape work: controllers/v2/types.ts (2825-line zod schema for all request/response types), scraper/scrapeURL/index.ts (1926-line pipeline), engines/index.ts (registry), and lib/error.ts + scrapeURL/error.ts for the TransportableError taxonomy.
- Middleware order is load-bearing: requestTiming → authMiddleware → countryCheck → checkCredits → blocklist → controller; reordering or omitting one (e.g. on a new route) silently drops auth, geo-restriction, billing or blocklist enforcement.
- Zero-data-retention must be threaded: ZDR suppresses OTel recording, GCS/index writes, and URL logging; any new code path that touches a scrape must check internalOptions.zeroDataRetention / getScrapeZDR before persisting or tracing.
- Tests require a running stack: `pnpm harness` spawns api+workers+containers; snips hit TEST_API_URL with TEST_API_KEY='test'/TEST_TEAM_ID='bypass' in self-host mode — do not hand-start the server, and scope test runs to the relevant suite since the full suite is slow.
- Naming traps: _addScrapeJobToBullMQ actually enqueues onto NuQ (BullMQ naming is vestigial); 'snips' means E2E tests; 'ACUC' is the auth/credit chunk; 'NuQ' queues live in Postgres tables prefixed nuq.; job 'mode' (not endpoint) selects worker behavior.
- Error contract: controllers return {success:false, error} with status codes mapped per TransportableError code in scrape.ts's catch block — new error codes need an explicit branch (e.g. SCRAPE_DNS_RESOLUTION_ERROR→200, SCRAPE_NO_CACHED_DATA→404) or they surface as UNKNOWN_ERROR 500s.
- Env/config changes: add the var to the Zod schema in config.ts (empty-string handling matters — use emptyStringAsUndefined), reference it via config.X, and update docker-compose.yaml common-env + CI workflow envs if the feature must work self-hosted.
- In-flight job compatibility: NuQ rows are JSONB payloads consumed by separate worker binaries, so changing ScrapeJobData/internalOptions shape requires backward-compat handling (the code already guards with ?? fallbacks like job.data.internalOptions?.orgId ?? null).