CodeVetter · Repo Unpack · Testing signals
ggml-org/llama.cpp
C++ tests live in tests/ and register with CTest via llama_build_and_test/llama_test helpers; coverage spans tokenizers (against models/ggml-vocab-*.gguf fixtures), grammars, chat templates/parsers, KV/recurrent state, samplers, quantization, and backend ops. The server has a parallel pytest suite (tools/server/tests) that spawns real llama-server processes, and the web UI has Vitest (unit/client/ui projects) plus Playwright e2e. CI runs ctest -L main and dedicated per-backend workflows.
- Test registration is CMake-driven: tests/CMakeLists.txt defines llama_build, llama_test, llama_test_cmd, llama_build_and_test helpers; most tests link llama-common (some also libllama) and are labeled 'main' or 'model' for ctest -L filtering.
- Tokenizer coverage is fixture-heavy: test-tokenizer-0 is built once and instantiated ~15 times against models/ggml-vocab-*.gguf files (bert-bge, command-r, deepseek-coder, gemma-4, gpt-2, llama-bpe/spm, qwen2, qwen35, ...); several slower tokenizer tests are commented out.
- Several tests are gated: Windows+shared-libs excludes tests using non-exported internals (test-unicode, test-sampling, test-grammar-*, test-chat, ...); test-opt is skipped under ASAN or GGML_SCHED_NO_REALLOC; backend-direct tests (test-barrier, test-quantize-*, test-rope) need non-DL CPU backend.
- A dummy-model fixture system exists: test-llama-archs generates tiny GGUFs for every architecture (test-generate-models ctest fixture), then test-recurrent-state-rollback and test-save-load-state run against generated qwen35-dense, nemotron_h-dense, deepseek4-moe, kimi-k3-moe models.
- A real model is downloaded in CI: test-download-model fetches tinyllamas/stories15M-q4_0.gguf (SHA256-pinned) via cmake/download-models.cmake as a ctest fixture required by test-thread-safety and test-state-restore-fragmented.
- Chat/template stack has dedicated tests: test-chat (links server-context lib), test-chat-template, test-chat-peg-parser, test-chat-auto-parser, test-jinja (plus a -py labeled variant), test-json-schema(-to-grammar), test-peg-parser with a suite of peg-parser/test-*.cpp files; test-chat-analysis is built but deliberately not registered (manual debug tool).
- Server has a pytest suite in tools/server/tests: tests.sh runs pytest -n auto --dist=worksteal with a 'slow' marker; utils.py ServerProcess spawns the real llama-server binary on a port with a tiny stories260K model; 27 unit files cover chat completion, OAI/Anthropic/GCP compat, embeddings, rerank, infill, lora, mcp, router, security, sleep, slots, speculative, streaming, vision, tool calls.
- Security tests exist for API keys: test_security.py asserts /health is public, static UI assets bypass auth, wrong/missing Bearer keys get 401 authentication_error, and a valid key returns 200.
- UI testing is three-tiered: vitest projects 'unit', 'client' (browser mode via playwright provider) and 'ui', plus 'test:e2e' playwright tests and storybook; scripts in tools/ui/package.json wire them into npm test.
- CI coverage: .github/workflows/server.yml runs the server pytest suite on PRs touching C/C++ or tools/server; build-cpu.yml runs 'ctest -L main --verbose --timeout 900'; separate workflows cover CUDA/Metal/Vulkan/SYCL/sanitizers/server-sanitize; python-lint and python-type-check gate the conversion code.
- Coverage gaps: test-backend-ops and test-tokenizer-1-bpe exist as build targets but have most of their test registrations commented out ('disabled due to slowness'); test-double-float is commented as SLOW; ggml backend compute correctness relies heavily on test-backend-ops which is built but only selectively registered.