CodeVetter · Repo Unpack · Extension points
firecrawl/firecrawl
The cleanest extension seams are the scrape engine registry (add an Engine string, handler, MRT and feature/quality entry), the transformer/postprocessor lists, and the route tables (v2Router wiring controllers through shared middlewares). NuQ job modes ('single_urls', 'kickoff', 'kickoff_sitemap') dispatch in processJobWithTracing, search providers chain in search/index.ts, threat-protection providers plug behind lib/threat-protection, and exchange eligibility is isolated in lib/exchange.ts. New env config goes in the Zod schema; new team flags ride TeamFlags in the ACUC.
- Adding a scrape engine = extend the Engine union, append to engines[] behind an env gate, and fill engineHandlers, engineMRTs and engineOptions (feature support + quality) — buildFallbackList and the waterfall pick it up automatically.
- Feature flags are declarative: the featureFlags tuple plus featureFlagOptions priorities drive which engines qualify; adding a flag requires updating the union, every engine's features map, and the format/options schema in controllers/v2/types.ts.
- New output formats plug in as transformers: executeTransformers runs modules like llmExtract, deterministicJson, diff (changeTracking), menu, product, audio, video, sendToSearchIndex; postprocessors (youtube) run after.
- Routes are registered flat in routes/v2.ts with per-route middleware composition; a new endpoint needs a controller under controllers/v2, a RateLimiterMode in types.ts, and the standard auth/credits/blocklist middleware stack.
- NuQ job dispatch is a mode switch: ScrapeJobData.mode ('single_urls'|'kickoff'|'kickoff_sitemap') selects processJob/processKickoffJob/processKickoffSitemapJob in processJobWithTracing — new job kinds extend the union in types.ts and add a branch.
- Search providers chain by config: search/index.ts tries fire_engine_search when FIRE_ENGINE_BETA_URL is set, then searxng_search when SEARXNG_ENDPOINT is set, then ddgSearch — a new provider slots into that fallback chain.
- URL-specific behavior is table-driven: urlSpecificParams overrides options per domain, engine-forcing (getEngineForUrl/initializeEngineForcing) pins engines per host, and the blocklist (initializeBlocklist/isUrlBlocked) is a warmed data file — all three are data-shaped extension points.
- Threat protection is provider-pluggable: lib/threat-protection resolves a policy per team/request and checkUrl consults providers (Google Web Risk normal/ZDR-synced modes, Zscaler ZIA), with org config exposed via /v2/team/threat-protection routes.
- Billing features are keyed by Autumn feature IDs (CREDITS_FEATURE_ID, SEARCH_CREDITS_FEATURE_ID, featureIdForBillingEndpoint) — new metered endpoints pass their featureId through checkCreditsMiddleware's second arg.
- SDKs are independently versioned packages under apps/*-sdk with per-language publish workflows; the API surface they consume is documented by openapi.json/v1-openapi.json/openapi-v0.json in apps/api.
- Config extension is centralized: new env vars require a Zod entry in configSchema (with helpers like emptyStringAsUndefined/delimitedList) and are then read as config.X everywhere; MCP action-log config even validates at startup via getMcpActionLogConfigErrors.