CodeVetter · Repo Unpack · Testing signals
langgenius/dify
Testing is layered and unusually heavy: ~1655 Python test files split into `unit_tests`, `integration_tests` (Docker-backed, CI-only), and `test_containers_integration_tests`, plus per-provider `vdb/*/tests` and `trace/*/tests` in the uv workspace. Web uses Vitest (+browser mode), e2e uses Cucumber+Playwright with accessibility (axe/WCAG) lanes, cli uses Vitest against a real Hono mock server, and the Go runtime has colocated `*_test.go`. CI shards API unit tests by historical durations and gates on codecov.
- `api/pytest.ini` configures coverage (`--cov=./api`, branch, json+xml reports), `--import-mode=importlib`, and `pytest-env` seeds dozens of fake provider keys plus `PLUGIN_DAEMON_URL`, `PLUGIN_DAEMON_KEY`, `INNER_API_KEY_FOR_PLUGIN`, and `MOCK_SWITCH=true` for hermetic unit runs.
- Backend tests live in `api/tests/` (1655 py files): `unit_tests/` mirrors the package tree (core, controllers, services, tasks, schedule, factories, migrations, tools, datasource, trigger, extension, app/layers/task_pipeline), `integration_tests/` covers controllers/services/workflow/model_runtime/tools/vdb/storage/tasks against real middleware, and `test_containers_integration_tests/` uses testcontainers.
- `make test` runs unit tests with `pytest -p no:benchmark --timeout 20 -n auto` over `api/tests/unit_tests` + all `api/providers/*/tests/unit_tests`, running `unit_tests/controllers` in a second pass with `--cov-append`; `make test-all` adds `--start-middleware` integration runs and `--start-vdb` VDB smoke tests (chroma, pgvector, qdrant, weaviate).
- CI `.github/workflows/api-tests.yml` downloads a shared timing baseline (`test-times.json` from a dify-test-infra stats repo) and runs `api/dev/pytest_sharding.py` to freeze per-shard file assignments, uploading the plan as an artifact for the shard jobs.
- A dedicated `.github/workflows/db-migration-test.yml` exists for migration correctness; codecov.yml at root configures coverage reporting (CODECOV_TOKEN secret plumbed through the reusable workflow).
- Web tests are Vitest with a browser project (`vitest.browser.setup.ts`, `vitest.setup.ts`, `vitest-browser-react`, `vitest-canvas-mock` deps); specs are colocated (`web/service/base.spec.ts`, `web/service/common.spec.ts`, `web/service/__tests__/`) and `pnpm test` = `vp test --project unit`.
- E2E is Cucumber+Playwright in `e2e/` (`cucumber.config.ts`, `features/{apps,auth,smoke,agent-v2,accessibility}`) with tag-selectable lanes including `@axe`/`@wcag-aa` accessibility scans and `@browser-smoke` Chromium+WebKit lanes; scripts orchestrate seed/reset/middleware/web-build lifecycle.
- CLI tests are Vitest colocated with commands (`cli/src/commands/**/*.test.ts`, `coverage.test.ts`, `agent-guides.test.ts` for meta-coverage) and run against a real Hono mock server at `cli/test/fixtures/dify-mock` — AGENTS.md explicitly forbids nock/msw/fetchMock.
- Go runtime tests are colocated (`internal/server/*_test.go`, `cmd/runner/main_test.go`, `packaging_test.go`) covering api/config/db/output/runtime/snapshot/tmux/service; landlock has build-split linux vs stub files suggesting sandbox behavior is only tested on Linux.
- Contract testing exists: `packages/contracts` holds `openapi-yaml.test.ts`, `service-api-contract.test.ts`, `knowledge-fs-contract.test.mjs`, `sandbox-contract.smoke.test.ts` validating generated TS clients against backend OpenAPI output; `api/dev/lint_response_contracts.py` (`make api-contract-lint`) lints Flask response docs against returned schemas.
- Static gates beyond unit tests: `make lint` runs ruff format+check --fix + response-contract lint + `lint-imports` (import-linter module boundaries) + `dotenv-linter` over `api/.env.example`/`web/.env.example`; `make type-check` runs pyrefly (`python-platform=linux`, `infer-with-first-use`) + mypy excluding tests/migrations.
- Integration tests are CI-owned per repo policy — `api/AGENTS.md` states docker-backed suites are not expected to run locally and agents must not start long-running services — so local verification is unit-test-only for most developers.
- Coverage gaps to note: web `typescript.ignoreBuildErrors: true` means type errors don't fail builds; the e2e suite is opt-in/tag-gated; `dify-agent` local tests live under `dify-agent/tests/local/` mirroring src with an explicit no-mock-for-real-boundaries rule. (inferred)