CodeVetter · Repo Unpack · Testing signals
Comfy-Org/ComfyUI
Testing is pytest-based and split into two trees: tests-unit (fast unit tests, run in CI on Linux/Windows/macOS with CPU torch) and tests (integration: execution tests that spawn a real server subprocess and drive it over WebSocket, and inference tests that run full workflows and compare generated images against a baseline directory). Markers 'inference' and 'execution' allow deselection. Coverage of the node library itself is thin — most tests target the execution engine, validation, caching, and server endpoints rather than model math.
- pytest.ini sets testpaths to both tests and tests-unit, registers 'inference' and 'execution' markers, and adds -s (no output capture); pythonpath=. makes the repo root importable.
- tests/execution/test_execution.py spins up the real server via subprocess.Popen([sys.executable, 'main.py', ...]), waits, connects a ComfyClient over websocket-client, submits prompt graphs built with GraphBuilder, and asserts on WS events and output images.
- tests/execution also covers async nodes, jobs API, preview method override, progress isolation, and the public API (test_public_api.py), with a testing_nodes helper package of custom nodes for the tests.
- tests/inference/test_inference.py runs real diffusion workflows (ComfyGraph wrapper, KSampler imported from comfy.samplers) writing images to --output_dir; tests/compare performs quality regression by comparing directories, and conftest.py deliberately runs test_quality last.
- tests-unit covers the execution engine (test_execution_list, validate_node_input, loop validation, nested loops, cache providers, reentry), server/user endpoints (prompt_server_test, system_user_endpoint_test), folder_paths, security (security_test dir), assets, and comfy internals — all CPU-only.
- CI: .github/workflows/test-unit.yml runs `pytest tests-unit` on ubuntu-latest/windows-2022/macos-latest with Python 3.12 and CPU-only torch; it is marked continue-on-error, so unit failures do not block merges.
- Additional workflows: test-execution.yml and test-ci.yml run the execution/integration suites, test-launch.yml smoke-tests server launch (--quick-test-for-ci exits after startup), and ruff.yml lints with the rule set pinned in pyproject.toml.
- Test deps are separate: tests-unit/requirements.txt plus websocket-client/opencv/scikit-image for integration tests per tests/README.md; inference tests require downloaded models, so they are not runnable in plain CI.
- No coverage tooling, no mocked-HTTP layer for aiohttp routes, and no tests under comfy/ for individual model implementations — model correctness is only exercised indirectly through the heavyweight inference suite. (inferred)
- A --quick-test-for-ci flag in main.py exits right after the worker thread starts, which is what test-launch.yml uses to verify the server boots cleanly.