# ggml-org/llama.cpp — codebase unpack

> Canonical page: https://codevetter.com/unpack/llama-cpp

This is a deterministic Repo Unpack scan of ggml-org/llama.cpp — a structural map, not a review, security audit, or maintainer-endorsed description.

- Scanned commit: 7ab4ee7baad2d920464cbacfad4f4b07cf111fd2 (2026-09-22)
- Last upstream commit: chat : Fix Muse Glimmer tool-call first parser error (#29242)
- Generated by codevetter 1.12.0 on 2026-09-22
- 3518 files scanned · 41.5 MB

## Languages

- C++: 752 files
- TypeScript: 407 files
- C: 387 files
- Svelte: 324 files
- Python: 222 files
- Markdown: 124 files
- YAML: 78 files
- Shell: 73 files

## Repository structure

- ggml/ — 1377 files
- tools/ — 1005 files
- src/ — 223 files
- examples/ — 197 files
- models/ — 102 files
- conversion/ — 94 files
- common/ — 93 files
- tests/ — 86 files

Probable source entrypoint: `src/` (inferred from the tree, not upstream docs)

## Package

- pyproject.toml: pytorch v0.0.0 — 8 declared dependencies

## Structural health signals

Average score 6.5/10 across 96 files; 50 hotspots. These are deterministic review leads, not proof of a bug.

## Recent history

- 7ab4ee7baad2 (2026-09-22) chat : Fix Muse Glimmer tool-call first parser error (#29242)
- 0ee9435b8f91 (2026-09-22) ci : publish snapdragon builds in release workflow (#29007)
- 8cfc315a8abd (2026-09-22) Add close button to UI toasts (#28246)
- ec5a12b85ae3 (2026-09-21) opencl: add A8 Q4_0 non-MoE dp4a binary kernel (#29055)
- c550d2f60bde (2026-09-22) ci : update Level Zero SDK to v1.33.1 and enable the L0/oneDNN CMake flags in the SYCL job (#29230)

## Overview

llama.cpp is a dependency-free C/C++ inference engine for LLMs and VLMs, built around the vendored ggml tensor library with pluggable hardware backends (CPU, CUDA, Metal, Vulkan, SYCL, RPC, and more). Its public surface is the C API in include/llama.h, consumed by a unified `llama` binary (app/llama.cpp) that dispatches subcommands like `serve`, `cli`, `quantize`, and `perplexity`. The flagship deliverable is llama-server: an OpenAI/Anthropic-compatible HTTP server with an embedded SvelteKit web UI, multi-slot continuous batching, a router mode that spawns child inference processes, MCP/tool-calling support, and multimodal input via the mtmd/CLIP library. Python tooling under conversion/, convert_hf_to_gguf.py and gguf-py handles Hugging Face to GGUF model conversion.

## System map

The repo is a CMake C/C++ monorepo layered as: ggml/ (tensor ops + hardware backend plugins, statically registered or dlopen'd), src/ (libllama: GGUF model loading, vocab/tokenizers, KV-cache memory, compute graph builders per architecture, samplers, public C API in include/llama.h), common/ (llama-common utility lib: arg parsing, chat templates via a custom Jinja engine, sampling orchestration, HF downloading, subprocess helpers), tools/ (user-facing binaries incl. server, cli, mtmd multimodal, quantize, perplexity, bench), app/ (unified `llama` binary), and Python conversion tooling (conversion/, gguf-py/, convert_hf_to_gguf.py). Tests live in tests/ (CTest C++ binaries) and tools/server/tests (pytest). The web UI is a SvelteKit app in tools/ui embedded into the server binary at build time.
- The unified `llama` binary is a thin dispatcher: app/llama.cpp defines a static command table `cmds[]` mapping subcommands (serve, cli, update, download, completion, bench, batched-bench, fit-params, quantize, perplexity, version, licenses, help) to entry functions like llama_server and llama_cli, and sets the LLAMA_APP_CMD env var before invoking them so router child processes re-invoke correctly. — [app/llama.cpp#L64-78](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L64-L78) [app/llama.cpp#L129-143](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L129-L143)
- The public API is a C ABI in include/llama.h (~1646 lines, ~246 LLAMA_API declarations) implemented primarily by src/llama.cpp, which forwards into internal classes llama_model, llama_context, llama_vocab, and llama_memory. — [include/llama.h](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/include/llama.h) [src/llama.cpp#L1-30](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L1-L30)
- Model instantiation is a giant architecture switch: llama_model_mapping() in src/llama-model.cpp maps each LLM_ARCH_* enum to a `new llama_model_<arch>(params)` subclass (e.g. llama_model_llama, llama_model_deepseek2); unknown archs throw 'unsupported model architecture'. — [src/llama-model.cpp#L43-373](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model.cpp#L43-L373)
- Per-architecture graph builders live in src/models/ (156 files, e.g. llama.cpp, deepseek2.cpp, qwen3.cpp) plugged into llama-graph.cpp which assembles ggml compute graphs executed via llama_context::decode(). — [src/llama-graph.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-graph.cpp) [src/llama-context.cpp#L4326-4335](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-context.cpp#L4326-L4335) [src/llama-graph.h#L101-363](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-graph.h#L101-L363)
- ggml is vendored in ggml/ with a backend registry: ggml_backend_registry's constructor statically registers compile-time backends (CUDA, Metal, SYCL, Vulkan, WebGPU, zDNN, VirtGPU, OpenCL, ZenDNN, Hexagon, CANN, BLAS, RPC, OpenVINO, ET, CPU) and ggml_backend_load()/ggml_backend_load_all() can dlopen backend shared libraries at runtime (GGML_BACKEND_DL). — [ggml/src/ggml-backend-reg.cpp#L115-174](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-reg.cpp#L115-L174) [ggml/src/ggml-backend-reg.cpp#L393](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-reg.cpp#L393) [ggml/src/ggml-backend-reg.cpp#L574-603](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-reg.cpp#L574-L603)
- common/ builds the llama-common static library holding shared tool infrastructure: common_params + arg parsing (arg.cpp, ~4.7k lines), chat template/tool-call handling (chat.cpp + jinja/ + parsers/), sampler orchestration (sampling.cpp), speculative decoding (speculative.cpp), HF/URL downloading (download.cpp, hf-cache.cpp), subprocess wrapper (subproc.cpp) and a cpp-httplib HTTP client helper (http.h). — [common/arg.cpp#L77-84](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L77-L84) [common/chat.h#L316-340](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/chat.h#L316-L340) [common/sampling.cpp#L187-210](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/sampling.cpp#L187-L210) [common/download.cpp#L45-90](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/download.cpp#L45-L90) [common/http.h#L100-118](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/http.h#L100-L118)
- tools/server implements the OpenAI/Anthropic-compatible HTTP server on cpp-httplib (vendor/cpp-httplib): server-http.cpp wraps httplib::Server/SSLServer, server.cpp registers all routes, server-context.cpp owns the inference slots/queue, and server-models.cpp implements router mode by spawning child llama-server subprocesses. — [tools/server/server-http.cpp#L7-22](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-http.cpp#L7-L22) [tools/server/server.cpp#L241-306](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L241-L306) [tools/server/server-models.cpp#L1163-1189](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L1163-L1189) [vendor/cpp-httplib](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/vendor/cpp-httplib)
- tools/cli is an interactive chat client that embeds the server: cli_context::init() spawns llama_server() on a std::thread bound to a random free port (cli_server::start) and all generation goes through HTTP SSE to /v1/chat/completions rather than calling libllama directly. — [tools/cli/cli-server.h#L33-56](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-server.h#L33-L56) [tools/cli/cli-context.cpp#L94-151](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L94-L151) [tools/cli/cli-context.cpp#L350-411](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L350-L411)
- Multimodal support is a separate mtmd library in tools/mtmd: clip.cpp (~6k lines) implements the vision encoder stack, mtmd-audio.cpp handles audio input, and mtmd_init_from_file() loads a separate --mmproj GGUF projector model alongside the main model. — [tools/mtmd/mtmd.cpp#L582](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/mtmd/mtmd.cpp#L582) [tools/mtmd/mtmd.cpp#L1085](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/mtmd/mtmd.cpp#L1085) [tools/mtmd/clip.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/mtmd/clip.cpp)
- Model conversion is Python: convert_hf_to_gguf.py drives per-architecture converter classes in conversion/ registered via @ModelBase.register, writing GGUF through the gguf-py package (gguf-py/pyproject.toml, name=gguf v0.19.0); tensor name/layout tables live in gguf-py/gguf/constants.py. — [convert_hf_to_gguf.py#L15-25](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/convert_hf_to_gguf.py#L15-L25) [conversion/base.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/conversion/base.py) [gguf-py/pyproject.toml](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/gguf-py/pyproject.toml) [docs/development/HOWTO-add-model.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/docs/development/HOWTO-add-model.md)
- The web UI is a SvelteKit + Vite + Tailwind app in tools/ui (package.json name=llama-ui) that is compiled to static assets and embedded into the server binary via a CMake-generated ui.cpp; LLAMA_USE_PREBUILT_UI downloads prebuilt assets from the HF bucket 'ggml-org/llama-ui' by default. — [tools/ui/package.json](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/package.json) [tools/ui/CMakeLists.txt#L1-37](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/CMakeLists.txt#L1-L37) [tools/server/server-http.cpp#L5](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-http.cpp#L5)
- Top-level CMakeLists.txt is the single build system (cmake 3.14...3.28): version is LLAMA_VERSION 0.4.1-dev, feature options include LLAMA_BUILD_COMMON/TESTS/TOOLS/EXAMPLES/SERVER/APP/UI, LLAMA_OPENSSL, LLAMA_SUBPROCESS, LLAMA_LLGUIDANCE, plus deprecated-option shims mapping LLAMA_CUDA etc. to GGML_* options. — [CMakeLists.txt#L1-23](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L1-L23) [CMakeLists.txt#L129-146](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L129-L146) [CMakeLists.txt#L174-195](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L174-L195)
- Third-party code is vendored under vendor/ (cpp-httplib, nlohmann json, miniaudio, stb, sheredom subprocess.h, hash) with a dedicated CI workflow .github/workflows/check-vendor.yml verifying vendor integrity. — [vendor/CMakeLists.txt](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/vendor/CMakeLists.txt) [.github/workflows/check-vendor.yml](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/.github/workflows/check-vendor.yml)
- Deployment shapes: static or shared libllama (BUILD_SHARED_LIBS default ON except MinGW/Emscripten), the `llama` unified binary, individual llama-* tool binaries, a WASM/Emscripten build (MEMORY64 enabled), Docker images via docs/docker.md, and an install-script self-update path gated on LLAMA_INSTALL_BUILD. — [CMakeLists.txt#L55-79](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L55-L79) [app/llama.cpp#L27-41](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L27-L41) [docs/build.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/docs/build.md)

## Feature catalog

User-facing surface splits into the `llama` unified binary subcommands, the llama-server HTTP API, the interactive `llama cli` chat client, offline analysis/conversion tools, and the embedded web UI. The server exposes OpenAI-compatible, Anthropic-compatible, Ollama-ish and legacy endpoints plus extras (rerank, infill, tokenize, slots save/load, LoRA hotswap, resumable streams, MCP tools). Feature scope for the server is deliberately bounded by tools/server/README-dev.md.
- llama-server registers a broad REST surface in llama_server(): /completion(s), /v1/completions, /chat/completions + /v1/chat/completions, /v1/responses, /v1/audio/transcriptions, /v1/messages + /v1/messages/count_tokens (Anthropic), /infill, /embedding(s) + /v1/embeddings, /rerank(ing) + /v1/rerank(ing), /tokenize, /detokenize, /apply-template, /chat/completions/input_tokens, /props, /models, /metrics, /health, /lora-adapters (GET/POST), /slots + /slots/:id_slot, and resumable-stream endpoints /v1/stream and /v1/streams/lookup. — [tools/server/server.cpp#L248-306](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L248-L306)
- Router mode activates when no model is specified: llama_server() rewires most route handlers to models_routes->proxy_post/proxy_get and adds /models/load, /models/unload, /models/sse and DELETE /models so models are loaded on demand into child processes. — [tools/server/server.cpp#L136-246](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L136-L246) [tools/server/server-models.cpp#L255-367](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L255-L367)
- `llama cli` provides an interactive REPL with slash commands /exit, /regen, /clear, /read <file>, /glob <pattern>, /image, /audio, /video, a banner showing model/ftype/modalities, multimodal staging via base64 data URLs, and optional --out-file logging of the session. — [tools/cli/cli-context.cpp#L413-456](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L413-L456) [tools/cli/cli-context.cpp#L517-555](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L517-L555) [tools/cli/cli-context.cpp#L306-341](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L306-L341)
- Hidden CLI subcommands still ship in the binary: completion (text completion), bench, batched-bench, fit-params (auto-size params to fit device memory), quantize, and perplexity — all marked hidden=true in the cmds[] table. — [app/llama.cpp#L64-78](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L64-L78) [tools/completion/completion.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/completion/completion.cpp) [tools/llama-bench/llama-bench.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/llama-bench/llama-bench.cpp) [tools/perplexity/perplexity.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/perplexity/perplexity.cpp)
- Speculative decoding is a first-class feature via common/speculative.cpp: draft-model, MTP (multi-token prediction) and other types are parsed by common_speculative_types_from_names / inferred from GGUF metadata, and the server tracks draft-acceptance metrics per slot. — [common/speculative.h#L8-30](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/speculative.h#L8-L30) [common/speculative.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/speculative.cpp) [tools/server/server-context.cpp#L4137-4139](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4137-L4139)
- Structured output is supported through GBNF grammars (sample grammars in grammars/), json-schema-to-grammar.cpp converting JSON Schema to GBNF, and optional llguidance integration (LLAMA_LLGUIDANCE cmake option, llama_sampler_init_llg). — [common/json-schema-to-grammar.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/json-schema-to-grammar.cpp) [common/sampling.cpp#L215](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/sampling.cpp#L215) [CMakeLists.txt#L146](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L146) [grammars/json.gbnf](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/grammars/json.gbnf)
- Chat formatting/parsing is handled by a dedicated Jinja engine (common/jinja, lexer+parser+runtime) plus a PEG-based parser framework (common/peg-parser.cpp, chat-peg-parser.cpp) and per-model tool-call parsers in common/parsers/ (kimi-k2, gpt-oss, ministral3, cohere2moe, etc.); common_chat_templates_init/apply produce the prompt and common_chat_parse extracts reasoning/tool calls. — [common/jinja/README.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/jinja/README.md) [common/chat.h#L316-340](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/chat.h#L316-L340) [common/parsers/kimi-k2.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/parsers/kimi-k2.cpp) [common/chat-peg-parser.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/chat-peg-parser.cpp)
- Model acquisition features: -hf <repo> Hugging Face downloads with an ETag-based file cache (hf-cache.cpp, write_etag/read_etag, atomic .tmp+rename writes), --model-url direct download, docker_repo support, and HF remote preset repos applied in router mode. — [common/download.cpp#L50-90](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/download.cpp#L50-L90) [common/hf-cache.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/hf-cache.cpp) [tools/server/server.cpp#L122-134](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L122-L134)
- Server-side agent features: --tools enables built-in tools including exec_shell_command, file read/write tools, and MCP server integration (--mcp-*) where MCP servers run as stdio subprocesses via common_subproc; disabled features return a 403 'feature_disabled' response. — [tools/server/server-tools.cpp#L1239-1247](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L1239-L1247) [tools/server/server-tools.cpp#L1960](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L1960) [tools/server/server-mcp.cpp#L340-351](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-mcp.cpp#L340-L351) [tools/server/server.cpp#L312-371](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L312-L371)
- KV/memory management features: multi-slot continuous batching with n_parallel slots, unified KV cache (--kv-unified / --kv-unified-per-slot auto-sizing), prompt cache in RAM (--cache-ram / cache_idle_slots), context checkpoints (--ctx-checkpoints), context shifting, and slot save/restore endpoints. — [tools/server/server.cpp#L162-172](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L162-L172) [tools/server/server-context.cpp#L1351-1370](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L1351-L1370) [tools/server/server-context.cpp#L3625-3635](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3625-L3635)
- LoRA adapters are loaded per request and hot-swappable: common_init_from_params loads params.lora_adapters via llama_adapter_lora_init, and the server exposes GET/POST /lora-adapters plus per-batch application via common_set_adapter_lora in update_slots(). — [common/common.cpp#L1343-1358](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/common.cpp#L1343-L1358) [tools/server/server.cpp#L284-285](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L284-L285) [tools/server/server-context.cpp#L2848-2865](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2848-L2865) [src/llama-adapter.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-adapter.cpp)
- Quantization and analysis tools: llama-quantize (tools/quantize), llama-imatrix (importance matrix), llama-perplexity (PPL/KLD eval), llama-gguf-split, llama-tokenize, llama-tts, llama-export-lora, llama-cvector-generator, and rpc-server exposing a remote compute backend. — [tools/quantize/quantize.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/quantize/quantize.cpp) [tools/imatrix/imatrix.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/imatrix/imatrix.cpp) [tools/rpc/rpc-server.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/rpc/rpc-server.cpp) [tools/gguf-split](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/gguf-split) [tools/tokenize](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/tokenize)
- Google Cloud Vertex AI compat: gcp_params in server-http.cpp reads AIP_MODE/AIP_HEALTH_ROUTE/AIP_PREDICT_ROUTE/AIP_HTTP_PORT env vars and remaps health/predict routes and port when AIP_MODE=PREDICTION. — [tools/server/server-http.cpp#L60-103](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-http.cpp#L60-L103) [tools/server/server.cpp#L308-309](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L308-L309)
- The web UI is a full chat frontend (SvelteKit static build embedded in the binary) with model management, MCP support, PWA assets, storybook stories, and its own Playwright e2e + Vitest suites configured in tools/ui/package.json. — [tools/ui/package.json#L6-27](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/package.json#L6-L27) [tools/ui/vite.config.ts](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/vite.config.ts) [tools/ui/playwright.config.ts](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/playwright.config.ts)

## Data flow

The canonical pipeline is: CLI args -> common_params (arg.cpp) -> model file resolution/download (download.cpp, hf-cache.cpp) -> GGUF parse + mmap via llama_model_loader -> per-arch llama_model subclass (tensors mapped into ggml backend buffers) -> llama_context with KV memory + graph builder -> llama_decode produces logits -> sampler chain produces tokens -> detokenize to text. In the server, HTTP JSON is tokenized into server_task objects flowing through server_queue -> server_slot -> a shared llama_batch -> llama_decode -> results streamed back as SSE.
- Argument handling flows through common_params_parser_init in common/arg.cpp, which builds a table of common_arg entries each supporting env-var fallback (LLAMA_ARG_* / LLAMA_ARG_NO_*), per-example gating (LLAMA_EXAMPLE_SERVER/CLI/MTMD/TTS), and preset-only flags. — [common/arg.cpp#L60-141](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L60-L141) [common/arg.cpp#L3379-3505](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L3379-L3505)
- Model loading: common_init_from_params -> llama_model_load_from_file -> llama_model_load_from_file_impl -> llama_model_load constructs llama_model_loader (GGUF metadata + tensor index, mmap/mlock/direct-IO via params.load_mode), calls llama_model_create for arch dispatch, then load_hparams, load_vocab, load_stats, load_tensors in that order. — [common/common.cpp#L1329-1398](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/common.cpp#L1329-L1398) [src/llama.cpp#L316-378](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L316-L378) [src/llama.cpp#L465-470](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L465-L470)
- GGUF input is parsed by ggml/src/gguf.cpp and wrapped by llama_model_loader which resolves multi-file splits (names like name-00002-of-00004.gguf) via llama_get_list_splits and provides typed metadata access through the GGUFMeta template layer. — [src/llama-model-loader.cpp#L80-120](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model-loader.cpp#L80-L120) [ggml/src/gguf.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/gguf.cpp)
- Device placement is decided in llama_prepare_model_devices: it partitions ggml_backend_dev_t devices into RPC servers (front of list), discrete GPUs, and iGPUs (only if no dGPU), dedupes by device_id, honors LLAMA_SPLIT_MODE_TENSOR by wrapping devices in a ggml_backend_meta_device, and trims to main_gpu for LLAMA_SPLIT_MODE_NONE. — [src/llama.cpp#L158-313](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L158-L313)
- Context creation in common_init_from_params loads LoRA adapters, initializes per-sequence samplers (common_sampler_init, optionally passed to the context for backend sampling via cparams.samplers), then calls llama_init_from_model which builds KV memory (llama_memory_i implementations: kv-cache, kv-cache-iswa, memory-hybrid, memory-recurrent) and the graph pipeline. — [common/common.cpp#L1343-1408](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/common.cpp#L1343-L1408) [src/llama-memory.h#L73-129](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-memory.h#L73-L129)
- Tokenization flows through llm_tokenizer implementations in src/llama-vocab.cpp (SPM, BPE with regex pre-tokenizers, UGM, etc.) selected by vocab type from GGUF metadata; unicode handling lives in src/unicode.cpp/unicode-data.cpp. — [src/llama-vocab.cpp#L73-286](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-vocab.cpp#L73-L286)
- Server request path: handle_completions_impl parses JSON, tokenizes via tokenize_input_prompts or process_mtmd_prompt (for multimodal), builds server_task objects with evaluated sampling params (server_schema::eval_llama_cmpl_schema), spawns child tasks when n_cmpl>1, and posts them to queue_tasks. — [tools/server/server-context.cpp#L4257-4342](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4257-L4342) [tools/server/server-task.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-task.cpp)
- Slot scheduling: process_single_task handles COMPLETION/INFILL/EMBEDDING/RERANK by picking a slot via get_available_slot (deferring to queue_tasks.defer when none free or when child tasks lack slots) then launch_slot_with_task / launch_slots_with_parent_task. — [tools/server/server-context.cpp#L2375-2453](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2375-L2453) [tools/server/server-queue.h#L53-77](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-queue.h#L53-L77)
- Decode loop: update_slots() runs pre_decode (context shift, checkpoint creation) -> batch.render() -> chunked decode() calls llama_decode(ctx_tgt, batch_view) inside queue_tasks.yield_to_queue (so metrics tasks can run mid-decode) -> post_decode per chunk; llama_decode itself is async so timings only settle after llama_synchronize. — [tools/server/server-context.cpp#L2793-2907](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2793-L2907) [tools/server/server-context.cpp#L3647-3686](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3647-L3686)
- Sampling is an ordered chain built by common_sampler_init: grammar/llg sampler first, then logit_bias, DRY, top-k, top-p, top-n-sigma, min-p, XTC, typical, temp_ext, infill, penalties, adaptive_p, and dist/temp at the end; chain order is data-driven from common_params_sampling. — [common/sampling.cpp#L187-404](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/sampling.cpp#L187-L404) [src/llama-sampler.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-sampler.cpp)
- Results flow back through server_response queues: slots emit server_task_result_cmpl_partial/final which the HTTP layer formats as OAI SSE (format_oai_sse), Anthropic SSE (format_anthropic_sse), or OAI responses SSE, terminating with 'data: [DONE]'; non-stream requests collect via rd.wait_for_all. — [tools/server/server-context.cpp#L4348-4523](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4348-L4523) [tools/server/server-queue.h#L162-238](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-queue.h#L162-L238)
- CLI data flow: cli_context keeps an OpenAI-style messages array; user input (plus staged base64 media parts) is POSTed to /v1/chat/completions with stream=true and timings_per_token, and SSE deltas are rendered by ui::assistant_turn into reasoning/content streams. — [tools/cli/cli-context.cpp#L287-411](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L287-L411)
- Conversion pipeline data flow: convert_hf_to_gguf.py reads HF config/tokenizer/safetensors via a per-arch ModelBase subclass, maps tensor names through gguf MODEL_TENSORS tables in gguf-py/gguf/constants.py, and writes metadata+tensors to a .gguf file consumable by llama_model_loader. (inferred) — [convert_hf_to_gguf.py#L1-40](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/convert_hf_to_gguf.py#L1-L40) [gguf-py/gguf/constants.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/gguf-py/gguf/constants.py) [conversion/base.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/conversion/base.py)

## Behavior traces

Three flows traced end-to-end: (1) `llama serve` startup including router vs single-model branches, (2) an HTTP chat completion through tokenize -> task queue -> slot -> decode -> SSE, and (3) `llama cli` startup which spawns an embedded server thread and streams chat over loopback HTTP.
- Server startup trace: app main() matches 'serve' -> llama_server(argc,argv) -> setlocale/SIGPIPE ignore -> common_init() -> server_stream_session_manager_start() -> common_params_parse(LLAMA_EXAMPLE_SERVER) -> llama_backend_init() + llama_numa_init() -> llama_server(params,...) which resolves HF presets, sizes KV pool, and constructs server_mcp, server_context, server_http_context. — [app/llama.cpp#L129-143](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L129-L143) [tools/server/server.cpp#L88-190](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L88-L190)
- Server startup (continued): ctx_http.init(params) builds the httplib (SSL)Server -> routes registered on ctx_http with ex_wrapper -> mcp_mgr.start + tools.setup -> model download via common_models_handler_apply -> ctx_http.start() (before model load so /health works) -> ctx_server.load_model(params) -> routes.update_meta -> ctx_server.start_loop() blocks the main thread in queue_tasks.start_loop(). — [tools/server/server.cpp#L192-246](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L192-L246) [tools/server/server.cpp#L383-493](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L383-L493) [tools/server/server.cpp#L540-558](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L540-L558)
- Inside load_model -> init(): wires queue callbacks (on_new_task -> process_single_task, on_update_slots -> update_slots, on_sleeping_state -> handle_sleeping_state), inits metrics, reads LLAMA_TRACE / LLAMA_SERVER_SLOTS_DEBUG / LLAMA_SERVER_SLOTS_N_DIFF env vars, sizes the shared llama_batch to max(n_batch, n_parallel), creates the RAM prompt cache, and initializes chat templates via common_chat_templates_init. — [tools/server/server-context.cpp#L1290-1460](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L1290-L1460)
- Chat completion trace: POST /v1/chat/completions -> routes.post_chat_completions -> handle_completions_impl -> tokenize/process_mtmd_prompt -> server_task(type=COMPLETION) per input -> rd.post_tasks -> server_queue::start_loop drains queue -> process_single_task -> get_available_slot -> launch_slot_with_task -> next update_slots() iteration batches it. — [tools/server/server-context.cpp#L4257-4342](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4257-L4342) [tools/server/server-queue.cpp#L278-316](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-queue.cpp#L278-L316) [tools/server/server-context.cpp#L2382-2453](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2382-L2453)
- Generation loop trace: update_slots() early-returns when all slots idle (else self-posts SERVER_TASK_TYPE_NEXT_RESPONSE) -> pre_decode() applies context-shift/checkpoints -> batch.render() -> for each n_batch chunk decode() -> queue_tasks.yield_to_queue wrapping llama_decode(ctx_tgt, batch_view) + llama_synchronize -> post_decode() samples and emits partial/final results. — [tools/server/server-context.cpp#L2807-2907](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2807-L2907) [tools/server/server-context.cpp#L3673-3686](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3673-L3686)
- Streaming response trace: first rd.next() result decides headers; then res->set_next drives chunked output — format_oai_sse/format_anthropic_sse/format_oai_resp_sse per result, SSE keepalive ping ':' when sse_ping_interval elapses, 'data: [DONE]' terminator for OAI formats; errors mid-stream are formatted as SSE error events. — [tools/server/server-context.cpp#L4380-4523](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4380-L4523)
- CLI startup trace: llama_cli -> common_init() -> common_params_parse(LLAMA_EXAMPLE_CLI) -> install SIGINT/SIGTERM handler -> cli_context(params) -> init(): either connect to --server-base or cli_server.start() spawns llama_server() on a std::thread with a random port, wait_ready() polls GET /health until 200, then client.wait_health and list_and_ask_models() (GET /v1/models) pick a model. — [tools/cli/cli.cpp#L33-64](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli.cpp#L33-L64) [tools/cli/cli-server.h#L34-84](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-server.h#L34-L84) [tools/cli/cli-context.cpp#L94-183](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L94-L183)
- CLI turn trace: run() prints banner -> user_turn.read_input() -> slash-command dispatch (/regen pops last message, /clear resets, /image|/audio|/video stages base64, /read and /glob inline file text) -> push_user_message -> generate_completion POSTs SSE to /v1/chat/completions -> delta.reasoning_content/delta.content rendered via ui::assistant_turn, timings pulled from chunk.timings. — [tools/cli/cli-context.cpp#L413-560](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L413-L560) [tools/cli/cli-context.cpp#L350-411](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/cli/cli-context.cpp#L350-L411)
- Model load trace (persistence path): llama_model_load_from_file -> llama_model_load_from_file_impl validates exactly one of metadata/path/file, installs a default progress callback -> llama_model_load -> llama_model_loader ctor (gguf parse, split resolution, mmap) -> llama_model_create(ml, params) -> llama_prepare_model_devices -> model->load_hparams -> load_vocab -> load_stats -> print_info -> load_tensors (supports vocab_only early return and -2 cancellation via progress callback). — [src/llama.cpp#L316-444](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L316-L444) [src/llama-model-loader.cpp#L80-105](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model-loader.cpp#L80-L105)
- Router child spawn trace: server_models::load (server-models.cpp) builds child argv, sets subprocess_option_no_window|combined_stdout_stderr, calls inst.subproc->sproc.create(child_args, options, child_env); a single waiter thread watches all children, parses 'cmd_child_to_router:state:' lines from stdout and calls on_child_exit on EOF/exit; request_child_exit writes 'cmd_router_to_child:exit' to the child's stdin. — [tools/server/server-models.cpp#L45-80](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L45-L80) [tools/server/server-models.cpp#L184-255](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L184-L255) [tools/server/server-models.cpp#L1163-1280](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L1163-L1280)
- Shutdown trace: SIGINT -> signal_handler -> shutdown_handler -> (single-model) mcp_mgr.shutdown() + ctx_server.terminate() which unblocks start_loop -> clean_up(): server_stream_session_manager_stop, ctx_http.stop, ctx_server.terminate, llama_backend_free; a second Ctrl+C force-exits with exit(1). — [tools/server/server.cpp#L25-49](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L25-L49) [tools/server/server.cpp#L448-558](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L448-L558)

## Testing signals

C++ tests live in tests/ and register with CTest via llama_build_and_test/llama_test helpers; coverage spans tokenizers (against models/ggml-vocab-*.gguf fixtures), grammars, chat templates/parsers, KV/recurrent state, samplers, quantization, and backend ops. The server has a parallel pytest suite (tools/server/tests) that spawns real llama-server processes, and the web UI has Vitest (unit/client/ui projects) plus Playwright e2e. CI runs ctest -L main and dedicated per-backend workflows.
- Test registration is CMake-driven: tests/CMakeLists.txt defines llama_build, llama_test, llama_test_cmd, llama_build_and_test helpers; most tests link llama-common (some also libllama) and are labeled 'main' or 'model' for ctest -L filtering. — [tests/CMakeLists.txt#L3-117](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L3-L117)
- Tokenizer coverage is fixture-heavy: test-tokenizer-0 is built once and instantiated ~15 times against models/ggml-vocab-*.gguf files (bert-bge, command-r, deepseek-coder, gemma-4, gpt-2, llama-bpe/spm, qwen2, qwen35, ...); several slower tokenizer tests are commented out. — [tests/CMakeLists.txt#L120-191](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L120-L191) [models/](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/models/)
- Several tests are gated: Windows+shared-libs excludes tests using non-exported internals (test-unicode, test-sampling, test-grammar-*, test-chat, ...); test-opt is skipped under ASAN or GGML_SCHED_NO_REALLOC; backend-direct tests (test-barrier, test-quantize-*, test-rope) need non-DL CPU backend. — [tests/CMakeLists.txt#L154-172](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L154-L172) [tests/CMakeLists.txt#L304-332](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L304-L332)
- A dummy-model fixture system exists: test-llama-archs generates tiny GGUFs for every architecture (test-generate-models ctest fixture), then test-recurrent-state-rollback and test-save-load-state run against generated qwen35-dense, nemotron_h-dense, deepseek4-moe, kimi-k3-moe models. — [tests/CMakeLists.txt#L195-256](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L195-L256) [tests/test-llama-archs.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/test-llama-archs.cpp)
- A real model is downloaded in CI: test-download-model fetches tinyllamas/stories15M-q4_0.gguf (SHA256-pinned) via cmake/download-models.cmake as a ctest fixture required by test-thread-safety and test-state-restore-fragmented. — [tests/CMakeLists.txt#L279-319](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L279-L319)
- Chat/template stack has dedicated tests: test-chat (links server-context lib), test-chat-template, test-chat-peg-parser, test-chat-auto-parser, test-jinja (plus a -py labeled variant), test-json-schema(-to-grammar), test-peg-parser with a suite of peg-parser/test-*.cpp files; test-chat-analysis is built but deliberately not registered (manual debug tool). — [tests/CMakeLists.txt#L163-166](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L163-L166) [tests/CMakeLists.txt#L258-277](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L258-L277)
- Server has a pytest suite in tools/server/tests: tests.sh runs pytest -n auto --dist=worksteal with a 'slow' marker; utils.py ServerProcess spawns the real llama-server binary on a port with a tiny stories260K model; 27 unit files cover chat completion, OAI/Anthropic/GCP compat, embeddings, rerank, infill, lora, mcp, router, security, sleep, slots, speculative, streaming, vision, tool calls. — [tools/server/tests/tests.sh](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/tests/tests.sh) [tools/server/tests/utils.py#L38-60](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/tests/utils.py#L38-L60) [tools/server/tests/unit/test_security.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/tests/unit/test_security.py)
- Security tests exist for API keys: test_security.py asserts /health is public, static UI assets bypass auth, wrong/missing Bearer keys get 401 authentication_error, and a valid key returns 200. — [tools/server/tests/unit/test_security.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/tests/unit/test_security.py)
- UI testing is three-tiered: vitest projects 'unit', 'client' (browser mode via playwright provider) and 'ui', plus 'test:e2e' playwright tests and storybook; scripts in tools/ui/package.json wire them into npm test. — [tools/ui/package.json#L17-24](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/package.json#L17-L24) [tools/ui/vite.config.ts](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/vite.config.ts) [tools/ui/playwright.config.ts](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/playwright.config.ts)
- CI coverage: .github/workflows/server.yml runs the server pytest suite on PRs touching C/C++ or tools/server; build-cpu.yml runs 'ctest -L main --verbose --timeout 900'; separate workflows cover CUDA/Metal/Vulkan/SYCL/sanitizers/server-sanitize; python-lint and python-type-check gate the conversion code. — [.github/workflows/server.yml#L1-40](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/.github/workflows/server.yml#L1-L40) [.github/workflows/build-cpu.yml#L127](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/.github/workflows/build-cpu.yml#L127) [.github/workflows/server-sanitize.yml](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/.github/workflows/server-sanitize.yml) [.github/workflows/python-lint.yml](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/.github/workflows/python-lint.yml)
- Coverage gaps: test-backend-ops and test-tokenizer-1-bpe exist as build targets but have most of their test registrations commented out ('disabled due to slowness'); test-double-float is commented as SLOW; ggml backend compute correctness relies heavily on test-backend-ops which is built but only selectively registered. — [tests/CMakeLists.txt#L177-193](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L177-L193) [tests/CMakeLists.txt#L310](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L310)

## Risk map

Biggest risks: server-side command execution features (exec_shell_command, MCP stdio subprocesses, router child spawning, CORS proxy) which are opt-in but powerful; unauthenticated-by-default HTTP API; remote content trust (HF downloads, remote presets, prebuilt UI assets); and a very large single-threaded decode loop in server-context.cpp (~5.5k lines) where a fatal error aborts all slots. GGML_ABORT/GGML_ASSERT turn many error paths into crashes rather than graceful failures.
- exec_shell_command is a built-in server tool that runs arbitrary shell commands via run_subprocess (sheredom subprocess.h, subprocess_option_search_user_path|inherit_environment); an 'isolate' mode can redirect file/exec ops through a container or remote host transport — gated behind --tools/--tools-runtime which the server logs as experimental security warnings. — [tools/server/server-tools.cpp#L187-207](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L187-L207) [tools/server/server-tools.cpp#L516-624](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L516-L624) [tools/server/server-tools.cpp#L1239-1247](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L1239-L1247) [tools/server/server.cpp#L350-380](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L350-L380)
- MCP integration spawns arbitrary configured commands as stdio subprocesses (server_mcp_stdio wraps common_subproc) with ndjson JSON-RPC over stdin/stdout; MCP server configs can be supplied as JSON including Cursor 'mcpServers' format. — [tools/server/server-mcp.cpp#L340-351](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-mcp.cpp#L340-L351) [tools/server/server-mcp.cpp#L132-176](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-mcp.cpp#L132-L176) [common/subproc.cpp#L30-49](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/subproc.cpp#L30-L49)
- Router mode spawns child llama-server processes with attacker-influenceable args (model params come from HTTP /models/load requests); it requires LLAMA_SUBPROCESS and throws when subprocess support is compiled out. — [tools/server/server-models.cpp#L1163-1189](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L1163-L1189) [tools/server/server-models.cpp#L1923-1924](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L1923-L1924) [CMakeLists.txt#L104-110](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L104-L110)
- Auth is optional: API keys come from --api-key/--api-key-file (LLAMA_ARG_API_KEY env); the server only warns (not fails) when cors_origins='*' with no keys, and prints 'do not expose to untrusted environments' warnings for router/tools/MCP features. — [common/arg.cpp#L3484-3505](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L3484-L3505) [tools/server/server.cpp#L324-341](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L324-L341) [tools/server/server.cpp#L373-380](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L373-L380)
- A /cors-proxy endpoint proxies arbitrary GET/POST requests when --ui-mcp-proxy is enabled (experimental, used by the Web UI for MCP), effectively an open proxy surface on the server port. — [tools/server/server.cpp#L328-341](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L328-L341) [tools/server/server-cors-proxy.h](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-cors-proxy.h)
- Remote model handling trusts network content: -hf downloads GGUFs and preset .ini files from HF (with an explicit 'only use presets that you can trust' warning), and LLAMA_USE_PREBUILT_UI pulls UI assets from a HF bucket at build time. — [tools/server/server.cpp#L520-524](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L520-L524) [common/download.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/download.cpp) [tools/ui/CMakeLists.txt#L1-37](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/CMakeLists.txt#L1-L37)
- `llama update` shells out: on LLAMA_INSTALL_BUILD it runs system('curl ... | sh') or powershell irm|iex against https://llama.app — a code-execution path dependent on TLS and remote server integrity. — [app/llama.cpp#L27-41](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L27-L41)
- Error handling frequently escalates to process abort: GGML_ABORT/GGML_ASSERT appear on hot paths (e.g. >3 consecutive empty decode batches aborts; model not implementing llama_model_base aborts), so malformed inputs or edge states can take down the whole server rather than one request. — [tools/server/server-context.cpp#L3652-3662](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3652-L3662) [src/llama.cpp#L332-335](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama.cpp#L332-L335) [tools/server/server-context.cpp#L4395-4398](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4395-L4398)
- Decode failure blast radius: when llama_decode returns nonzero, decode() sends errors to ALL processing slots and clears the entire context per-slot ('it's complicated to keep track of how much of the current batch has been processed'), so one bad request can kill concurrent sessions. — [tools/server/server-context.cpp#L3688-3719](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3688-L3719)
- server-context.cpp is a 5,558-line pimpl monolith owning slots, batching, checkpointing, prompt cache, metrics, LoRA, speculative decode and all route handlers — high coupling makes regressions likely; code comments admit 'alora handling is too messy, need to refactor'. — [tools/server/server-context.cpp#L2853](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2853) [tools/server/server-context.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp)
- GGUF/mmap input parsing is a trust boundary: llama_model_loader and gguf.cpp parse arbitrary model files (metadata strings, tensor offsets) and mmap them directly; params.check_tensors exists but untrusted model files remain a memory-safety attack surface. (inferred) — [src/llama-model-loader.cpp#L1-30](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model-loader.cpp#L1-L30) [src/llama-mmap.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-mmap.cpp) [ggml/src/gguf.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/gguf.cpp)
- SSL is optional and off unless CPPHTTPLIB_OPENSSL_SUPPORT is compiled in AND both --ssl-key-file/--ssl-cert-file are given; otherwise the API (including Bearer keys) runs plaintext HTTP — default port :8080 with a pending change to :9931. — [tools/server/server-http.cpp#L107-124](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-http.cpp#L107-L124) [CMakeLists.txt#L144](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L144) [tools/server/server.cpp#L514-518](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L514-L518)
- Debug/env side-channels: LLAMA_TRACE, LLAMA_SERVER_SLOTS_DEBUG, LLAMA_SERVER_SLOTS_N_DIFF change runtime behavior; --prompts-log-dir writes every prompt to disk (path_prompts_log_dir); prompts are also logged at debug level. — [tools/server/server-context.cpp#L1316-1341](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L1316-L1341) [tools/server/server-context.cpp#L4281-4289](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L4281-L4289)

## Extension points

Primary extension seams: new model architectures (conversion/ ModelBase.register + src/models/*.cpp + llama_model_mapping switch + gguf constants), new ggml backends (ggml_backend_reg_t + dlopen registry), new samplers (llama_sampler interface + common_sampler_init chain), new chat/tool parsers (common/parsers + PEG/jinja), new CLI args (common_arg table), new server endpoints (ctx_http.get/post + server_routes handlers), new built-in tools (server_tool subclasses), and new unified-binary subcommands (cmds[] table).
- New model architecture recipe is documented: register a TextModel/MmprojModel subclass via @ModelBase.register in conversion/, add MODEL_ARCH + MODEL_TENSORS entries in gguf-py/gguf/constants.py, add LLM_ARCH_* enum, a src/models/<arch>.cpp graph builder, and a case in llama_model_mapping — see docs/development/HOWTO-add-model.md. — [docs/development/HOWTO-add-model.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/docs/development/HOWTO-add-model.md) [src/llama-model.cpp#L43-373](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model.cpp#L43-L373) [gguf-py/gguf/constants.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/gguf-py/gguf/constants.py) [conversion/base.py](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/conversion/base.py)
- ggml backends are plugins: a backend exports a ggml_backend_*_reg() registered statically under #ifdef GGML_USE_<X> or loaded at runtime by ggml_backend_load(path) which dlopens a shared library — GGML_BACKEND_DL builds each backend as a separate .so. — [ggml/src/ggml-backend-reg.cpp#L115-200](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-reg.cpp#L115-L200) [ggml/src/ggml-backend-reg.cpp#L393](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-reg.cpp#L393) [ggml/src/ggml-backend-dl.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/ggml/src/ggml-backend-dl.cpp)
- Samplers implement the llama_sampler interface (src/llama-sampler.cpp provides ~20 stock samplers); common_sampler_init assembles them into a chain in a data-driven order, so adding a sampler means a new llama_sampler_init_* plus a slot in the ordering switch. — [src/llama-sampler.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-sampler.cpp) [common/sampling.cpp#L187-404](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/sampling.cpp#L187-L404)
- Chat/tool-call parsing is extensible at three layers: the standalone jinja engine (common/jinja) for templates, per-model PEG parsers in common/parsers/*.cpp (one file per model family), and chat-auto-parser which auto-detects model-specific features — all surfaced via common_chat_templates_apply/common_chat_parse. — [common/chat.h#L316-340](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/chat.h#L316-L340) [common/parsers/kimi-k2.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/parsers/kimi-k2.cpp) [common/chat-auto-parser.h](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/chat-auto-parser.h) [common/jinja/README.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/jinja/README.md)
- CLI arguments are declarative: each option is a common_arg entry in common_params_parser_init with flags, env var (set_env), applicable examples (set_examples/set_excludes), and handler lambda; get_common_arg_defs exposes the table for introspection. — [common/arg.cpp#L77-141](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L77-L141) [common/arg.cpp#L3379-3420](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L3379-L3420)
- Server endpoints plug in via ctx_http.get/post/del + a handler on server_routes (routes registered in llama_server); all handlers are wrapped by ex_wrapper which maps std::invalid_argument to 400 and other exceptions to 500 with an OAI-style error body. — [tools/server/server.cpp#L54-86](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L54-L86) [tools/server/server.cpp#L248-306](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L248-L306)
- Built-in server tools extend the server_tool interface (name/schema/run) and are appended in the tools registry (tools.push_back(std::make_unique<server_tool_...>())); an isolate-backed tools_io abstracts file/exec ops for sandboxed runtimes. — [tools/server/server-tools.cpp#L149-207](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L149-L207) [tools/server/server-tools.cpp#L516-624](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L516-L624) [tools/server/server-tools.cpp#L1960](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-tools.cpp#L1960)
- New unified-binary subcommands: add an entry to the cmds[] table in app/llama.cpp (name, desc, aliases, hidden flag, func) — hidden commands exist but aren't listed in default help. — [app/llama.cpp#L49-78](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L49-L78)
- KV/memory is abstracted behind llama_memory_i (seq_rm/seq_cp/seq_keep/get_can_shift/apply); multiple implementations already coexist (llama-kv-cache, -iswa, -dsa, -dsv4, -msa, memory-hybrid, memory-recurrent) selected per architecture. — [src/llama-memory.h#L73-129](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-memory.h#L73-L129) [src/llama-kv-cache.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-kv-cache.cpp) [src/llama-memory-recurrent.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-memory-recurrent.cpp)
- Structured output plugs through grammars: standalone .gbnf files in grammars/, JSON-Schema->GBNF conversion, and an optional llguidance backend (LLAMA_LLGUIDANCE) via llama_sampler_init_llg in the sampler chain. — [grammars/README.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/grammars/README.md) [common/json-schema-to-grammar.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/json-schema-to-grammar.cpp) [common/sampling.cpp#L215](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/sampling.cpp#L215)
- Speculative decoding types are enumerable and extensible: common_speculative_type_from_name / _from_gguf map names and GGUF metadata to enum common_speculative_type (draft, MTP, ...), implemented in common/speculative.cpp (~3k lines). — [common/speculative.h#L8-40](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/speculative.h#L8-L40) [common/speculative.cpp](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/speculative.cpp)
- MCP servers extend tool surface without code changes: server_mcp_server_config::parse_from_json accepts Cursor-style 'mcpServers' JSON to spawn stdio MCP servers whose tools are listed via list_tools and invoked via call_tool. — [tools/server/server-mcp.cpp#L132-176](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-mcp.cpp#L132-L176) [tools/server/server-mcp.cpp#L283-334](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-mcp.cpp#L283-L334)

## Contributor handoff

Read AGENTS.md first — it is strict: ASCII-only, concise comments, no new test files without maintainer approval, never commit/push/PR on the user's behalf, reuse existing infra. Build with cmake -B build && cmake --build build -j; run C++ tests via ctest and server tests via tools/server/tests/tests.sh. Key files to read before editing: common/arg.cpp (params), tools/server/README-dev.md (server feature scope), docs/development/HOWTO-add-model.md (arch additions), and the pimpl of whatever you touch (server_context_impl, llama_model subclasses).
- AGENTS.md is mandatory and unusual: it forbids agents from writing PR descriptions/comments/reviewer responses, forbids git push/gh pr create entirely (project ban risk), requires 'Assisted-by:' not 'Co-authored-by:' in human-approved commits, and tells fully autonomous agents to stop and exclude llama.cpp from contribution targets. — [AGENTS.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/AGENTS.md)
- Style rules enforced by maintainers: no emdashes/unicode arrows (use ASCII '-', '->', 'x', '...'), comments are 1-2 lines max in ASD-STE100 simplified English, no hard-wrapping comments, no comment-first-then-code, and do NOT split a line mid-sentence to fit a column width. — [AGENTS.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/AGENTS.md)
- Do not add new files under tests/ without maintainer approval — reuse existing test infrastructure; test-chat links against the server-context library so server chat changes can be exercised from tests/test-chat.cpp. — [AGENTS.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/AGENTS.md) [tests/CMakeLists.txt#L163-166](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L163-L166)
- Server feature scope is contractually bounded by tools/server/README-dev.md: in-scope = inference, chat/tool calling, API compat, multimodal, memory mgmt, Web UI needs; out-of-scope = server-side agentic loops, internal-state APIs, model-specific features; file-IO features (MCP, model save/load) must be disabled by default for security. — [tools/server/README-dev.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/README-dev.md)
- Build: cmake -B build (Release default, BUILD_SHARED_LIBS on) then cmake --build build -j produces bin/llama plus llama-* tools; LLAMA_BUILD_UI embeds the web UI; LLAMA_SUBPROCESS is off on iOS/Android/Emscripten which disables server router mode and server tools. — [CMakeLists.txt#L104-146](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/CMakeLists.txt#L104-L146) [tools/server/server-models.cpp#L1923-1924](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-models.cpp#L1923-L1924)
- Test invocation: ctest --test-dir build (label 'main' runs the core suite; 'model' needs downloaded models); server suite: cd tools/server/tests && ./tests.sh (needs PYTEST deps, spawns built server); UI: cd tools/ui && npm test (vitest+playwright). — [tests/CMakeLists.txt#L26-47](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L26-L47) [tools/server/tests/tests.sh](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/tests/tests.sh) [tools/ui/package.json#L17-24](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/ui/package.json#L17-L24)
- Before touching CLI behavior read common/arg.cpp's common_params_parser_init — every flag is a common_arg with examples/env/handler; adding a flag means one table entry plus documentation regeneration; LLAMA_EXAMPLE_* gating controls which binaries accept the flag. — [common/arg.cpp#L77-141](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.cpp#L77-L141) [common/arg.h](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/arg.h)
- Server edits: the inference core is server_context_impl inside tools/server/server-context.cpp (pimpl over server_context); slot scheduling lives in process_single_task/get_available_slot/launch_slot_with_task; decode loop is update_slots -> pre_decode/decode/post_decode; HTTP routes are registered in server.cpp not in server-context. — [tools/server/server-context.cpp#L2375-2453](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2375-L2453) [tools/server/server-context.cpp#L2793-2907](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2793-L2907) [tools/server/server.cpp#L248-306](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L248-L306)
- Adding a model arch requires coordinated edits across five places: conversion/<arch>.py (ModelBase.register), gguf-py/gguf/constants.py (MODEL_ARCH/MODEL_TENSORS), src/llama-arch.* (LLM_ARCH_*), src/models/<arch>.cpp (graph builder), and llama_model_mapping in src/llama-model.cpp — HOWTO-add-model.md is the canonical guide. — [docs/development/HOWTO-add-model.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/docs/development/HOWTO-add-model.md) [src/llama-model.cpp#L43-373](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-model.cpp#L43-L373) [src/llama-arch.h](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/src/llama-arch.h)
- The chat stack has traps: llama.cpp uses its own Jinja engine in common/jinja (NOT minja — AGENTS.md explicitly warns about stale knowledge); tool-call parsers are PEG-based in common/peg-parser.cpp + common/parsers; use test-chat-peg-parser/test-jinja to validate changes. — [AGENTS.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/AGENTS.md) [common/jinja/README.md](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/common/jinja/README.md) [tests/CMakeLists.txt#L258-263](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tests/CMakeLists.txt#L258-L263)
- Concurrency model: a single decode thread runs queue_tasks.start_loop -> update_slots; HTTP handlers run on httplib threads and only communicate via queue_tasks.post + queue_results; decode yields via yield_to_queue only for metrics/slot-read tasks — never call llama_decode off the main loop. — [tools/server/server-queue.h#L14-45](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-queue.h#L14-L45) [tools/server/server-context.cpp#L2375-2380](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L2375-L2380) [tools/server/server-queue.cpp#L278-316](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-queue.cpp#L278-L316)
- Known foot-guns: GGML_ABORT crashes on error paths; llama_decode errors clear entire slots; llama-server.cpp default port 8080 is slated to change to 9931; embeddings require n_batch<=n_ubatch (auto-clamped with warning); 'completion' is the legacy hidden tool — prefer server/cli paths for new work. — [tools/server/server-context.cpp#L3652-3719](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server-context.cpp#L3652-L3719) [tools/server/server.cpp#L148-152](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L148-L152) [tools/server/server.cpp#L514-518](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/tools/server/server.cpp#L514-L518) [app/llama.cpp#L69](https://github.com/ggml-org/llama.cpp/blob/7ab4ee7baad2d920464cbacfad4f4b07cf111fd2/app/llama.cpp#L69)

Source repository: https://github.com/ggml-org/llama.cpp
Full evidence corpus: https://github.com/Codevetter/codevetter/tree/main/benchmarks/repo-unpacks/llama-cpp.json

## Public product links

- [CodeVetter](https://codevetter.com/)
- [Download](https://codevetter.com/download)
- [Documentation](https://codevetter.com/docs/)
- [Source](https://github.com/Codevetter/codevetter)