CodeVetter · Repo Unpack · Testing signals
langflow-ai/langflow
Backend tests are pytest with asyncio_mode=auto across three roots (src/backend/tests ~778 files, src/lfx/tests ~425, src/bundles/*/tests ~76), with a rich marker taxonomy and heavyweight plugins (blockbuster for blocking-call detection, pytest-timeout in thread/faulthandler mode, xdist, rerunfailures, flakefinder, hypothesis, pyleak). The conftest boots the real app via create_app/LifespanManager over ASGITransport and StaticPool SQLite. Frontend uses Jest unit tests plus Playwright e2e (a11y/core/extended/live suites) with merged nyc coverage. CI runs a Python matrix (3.10/3.14) with split groups, bundle-guarded jobs, template tests, docker tests, CodeQL, and DB-migration validation.
- pytest is configured at the repo root: testpaths = src/backend/tests, src/lfx/tests, src/bundles/*/tests; asyncio_mode=auto; per-test timeout=90s with timeout_method=thread (temporary diagnostic for a Group-3 hang that dumps all thread stacks); addopts disables the benchmark plugin.
- Markers declared: async_test, api_key_required (deselected in CI via `-m "not api_key_required"` — those tests hit live provider APIs), no_blockbuster, benchmark, unit, integration, slow, security (IDOR/auth/access-control regression), real_services (needs real SQLite+Postgres+Redis).
- Dev-group test tooling is extensive: blockbuster (blocking-call detection in async), pytest-xdist/-sugar/-instafail/-cov/-split/-flakefinder/-rerunfailures/-timeout/-profiling, respx (httpx mocks), hypothesis, pyleak (leak detection), locust (load), fakeredis, codeflash, and mcp-server-fetch.
- The backend conftest builds a real app: TestClient/AsyncClient over ASGITransport + asgi-lifespan LifespanManager around create_app(), a StaticPool async engine for SQLite, blockbuster_ctx, and api-key fixtures; tests are instructed to prefer real integrations over mocks.
- Component tests have dedicated base classes (ComponentTestBaseWithClient / ComponentTestBaseWithoutClient) requiring component_class, default_kwargs, file_names_mapping fixtures, living under src/backend/tests/unit/components.
- Frontend testing: Jest (`npm test`) plus Playwright e2e under src/frontend/tests (a11y, core, extended, live, assets, fixtures), jest coverage and playwright-coverage merged via nyc and uploaded to Codecov; a11y-scan and a11y-unit-tests workflows exist.
- CI Python job matrixes Python 3.10 and 3.14, splits unit tests into groups via `make unit_tests args="--splits N --group G --reruns 5 --cov"`, and a separate job installs the bundles extra and runs only tests guarded by pytest.importorskip('lfx_bundles').
- Specialized CI: cross-platform-test, cross-bundle-test (bundle dependency range resolution), db-migration-validation, template-tests (runs test_starter_projects.py against all starter templates), docker_test/docker builds, CodeQL, smoke-tests, and stress-tests workflows.
- Integration tests are segregated under src/backend/tests/integration (coverage config ignores them: `--ignore=tests/integration`); there are also locust load tests (src/backend/tests/locust), a stress dir exempted from normal lint rules, and a `regressions/` folder of per-version YAML specs (1.9.x-1.12.x).
- Known test fragility is documented: DB tests 'may fail in batch but pass individually'; tests inside sub-packages need `uv sync --group dev --package <pkg>` first or dev deps like fakeredis are missing; tests set LANGFLOW_MODELS_DEV_REFRESH=false because the background fetch hits the network and trips pyleak.