CodeVetter · Repo Unpack · Testing signals
firecrawl/firecrawl
Tests run under Vitest (vitest.config.ts, globals on, 120s timeouts, junit output) with ~112 test files under src/__tests__ plus colocated *.test.ts units. The dominant pattern is 'snips' — end-to-end tests hitting a live API brought up by `pnpm harness` — gated by env flags (TEST_SUITE_SELF_HOSTED, OPENAI_API_KEY, SEARXNG_ENDPOINT, PROXY_SERVER). CI (test-server.yml) runs a matrix over engine/proxy/search/ai/nuq-backend on self-hosted infra. Coverage is strong on the scrape pipeline and route surface; deep unit coverage exists for config-adjacent libs but many services/ workers have no dedicated tests.
- Test runner is Vitest with globals:true, environment node, include src/**/*.test.ts, testTimeout/hookTimeout 120s, junit reporter to test-results/junit.xml; a resolve alias redirects pdf-parse to skip its module.parent debug self-test.
- package.json test scripts split the suite by auth mode: test (exclude e2e_noAuth), test:local-no-auth (exclude e2e_withAuth), test:full, test:prod, and test:snips scoped to src/__tests__/snips/v1 and v2.
- Snips are true E2E: __tests__/snips/lib.ts derives TEST_API_URL/TEST_SUITE_WEBSITE from config, exposes describeIf/testIf gating helpers and capability flags (HAS_FIRE_ENGINE, HAS_PLAYWRIGHT, HAS_PROXY, HAS_AI), and refuses local TEST_SUITE_WEBSITE in production mode.
- The snips/v2 directory alone holds ~40 suites covering scrape, crawl, batch, map, parse, parsers/pdf variants, billing, concurrency, keyless, key-restriction, ip-restriction, idempotency (idmux), browser replay, monitors, agent auth/threads/effort, alexandria feedback and Express 5 behavior.
- AGENTS.md prescribes the workflow: write E2E 'snips' (1 happy + 1+ failure path), always use scrapeTimeout from ./lib, gate fire-engine tests with !TEST_SUITE_SELF_HOSTED and AI tests additionally on OPENAI_API_KEY/OLLAMA_BASE_URL, then run via `pnpm harness jest` (never pnpm start manually).
- CI test-server.yml runs a self-hosted matrix (playwright|fetch × proxy|no-proxy × searxng × openai × nuq postgres, plus one fdb-backend leg) with redis/rabbitmq services, installing FoundationDB client and setting NUQ_DATABASE_URL/NUQ_RABBITMQ_URL/USE_GO_MARKDOWN_PARSER/ALLOW_LOCAL_WEBHOOKS.
- Each SDK has a dedicated publish/test workflow (test-js-sdk, test-go-sdk, test-rust-sdk, test-dotnet-sdk, etc.) and repo-level workflows cover deploy-image staging/prod, nuq-postgres deploy, playwright deploy, redis deploy, scrape-evals and npm-audit remediation.
- Unit tests exist colocated for critical libs: exchange.test.ts, canonical-url.test.ts, crawl-redis.test.ts, job-state-store.test.ts, key-restriction.test.ts, image-ocr-gate.test.ts, ip-restriction.test.ts, oauth-token-introspection.test.ts, rate-limiter.test.ts, scrapeURL.test.ts and snips mocks under __tests__/snips/mocks.
- Coverage gaps: worker orchestration files (queue-worker.ts, nuq.ts, nuq-worker-runner.ts, harness.ts) and most services/monitoring, webhook delivery internals, and the go-html-to-md-service (only handler_test.go) have thin or no direct unit tests — behavior is mostly exercised through snips. (inferred)
- A dedicated test fixture app (apps/test-site, name firecrawl-test-site) and TEST_SUITE_WEBSITE=http://127.0.0.1:4321 in CI provide deterministic scrape targets; knip (knip.config.ts) plus a pre-commit hook enforces no unused exports/files.