ggml-org/llama.cpp

Overview

llama.cpp is a dependency-free C/C++ inference engine for LLMs and VLMs, built around the vendored ggml tensor library with pluggable hardware backends (CPU, CUDA, Metal, Vulkan, SYCL, RPC, and more). Its public surface is the C API in include/llama.h, consumed by a unified `llama` binary (app/llama.cpp) that dispatches subcommands like `serve`, `cli`, `quantize`, and `perplexity`. The flagship deliverable is llama-server: an OpenAI/Anthropic-compatible HTTP server with an embedded SvelteKit web UI, multi-slot continuous batching, a router mode that spawns child inference processes, MCP/tool-calling support, and multimodal input via the mtmd/CLIP library. Python tooling under conversion/, convert_hf_to_gguf.py and gguf-py handles Hugging Face to GGUF model conversion.

System map

The repo is a CMake C/C++ monorepo layered as: ggml/ (tensor ops + hardware backend plugins, statically registered or dlopen'd), src/ (libllama: GGUF model loading, vocab/tokenizers, KV-cache memory, compute graph builders per architecture, samplers, public C API in include/llama.h), common/ (llama-common utility lib: arg parsing, chat templates via a custom Jinja engine, sampling orchestration, HF downloading, subprocess helpers), tools/ (user-facing binaries incl. server, cli, mtmd multimodal, quantize, perplexity, bench), app/ (unified `llama` binary), and Python conversion tooling (conversion/, gguf-py/, convert_hf_to_gguf.py). Tests live in tests/ (CTest C++ binaries) and tools/server/tests (pytest). The web UI is a SvelteKit app in tools/ui embedded into the server binary at build time.

• The unified `llama` binary is a thin dispatcher: app/llama.cpp defines a static command table `cmds[]` mapping subcommands (serve, cli, update, download, completion, bench, batched-bench, fit-params, quantize, perplexity, version, licenses, help) to entry functions like llama_server and llama_cli, and sets the LLAMA_APP_CMD env var before invoking them so router child processes re-invoke correctly.
  app/llama.cpp#L64-78app/llama.cpp#L129-143
• The public API is a C ABI in include/llama.h (~1646 lines, ~246 LLAMA_API declarations) implemented primarily by src/llama.cpp, which forwards into internal classes llama_model, llama_context, llama_vocab, and llama_memory.
  include/llama.hsrc/llama.cpp#L1-30
• Model instantiation is a giant architecture switch: llama_model_mapping() in src/llama-model.cpp maps each LLM_ARCH_* enum to a `new llama_model_<arch>(params)` subclass (e.g. llama_model_llama, llama_model_deepseek2); unknown archs throw 'unsupported model architecture'.
  src/llama-model.cpp#L43-373
• Per-architecture graph builders live in src/models/ (156 files, e.g. llama.cpp, deepseek2.cpp, qwen3.cpp) plugged into llama-graph.cpp which assembles ggml compute graphs executed via llama_context::decode().
  src/llama-graph.cppsrc/llama-context.cpp#L4326-4335src/llama-graph.h#L101-363
• ggml is vendored in ggml/ with a backend registry: ggml_backend_registry's constructor statically registers compile-time backends (CUDA, Metal, SYCL, Vulkan, WebGPU, zDNN, VirtGPU, OpenCL, ZenDNN, Hexagon, CANN, BLAS, RPC, OpenVINO, ET, CPU) and ggml_backend_load()/ggml_backend_load_all() can dlopen backend shared libraries at runtime (GGML_BACKEND_DL).
  ggml/src/ggml-backend-reg.cpp#L115-174ggml/src/ggml-backend-reg.cpp#L393ggml/src/ggml-backend-reg.cpp#L574-603
• common/ builds the llama-common static library holding shared tool infrastructure: common_params + arg parsing (arg.cpp, ~4.7k lines), chat template/tool-call handling (chat.cpp + jinja/ + parsers/), sampler orchestration (sampling.cpp), speculative decoding (speculative.cpp), HF/URL downloading (download.cpp, hf-cache.cpp), subprocess wrapper (subproc.cpp) and a cpp-httplib HTTP client helper (http.h).
  common/arg.cpp#L77-84common/chat.h#L316-340common/sampling.cpp#L187-210common/download.cpp#L45-90common/http.h#L100-118
• tools/server implements the OpenAI/Anthropic-compatible HTTP server on cpp-httplib (vendor/cpp-httplib): server-http.cpp wraps httplib::Server/SSLServer, server.cpp registers all routes, server-context.cpp owns the inference slots/queue, and server-models.cpp implements router mode by spawning child llama-server subprocesses.
  tools/server/server-http.cpp#L7-22tools/server/server.cpp#L241-306tools/server/server-models.cpp#L1163-1189vendor/cpp-httplib
• tools/cli is an interactive chat client that embeds the server: cli_context::init() spawns llama_server() on a std::thread bound to a random free port (cli_server::start) and all generation goes through HTTP SSE to /v1/chat/completions rather than calling libllama directly.
  tools/cli/cli-server.h#L33-56tools/cli/cli-context.cpp#L94-151tools/cli/cli-context.cpp#L350-411
• Multimodal support is a separate mtmd library in tools/mtmd: clip.cpp (~6k lines) implements the vision encoder stack, mtmd-audio.cpp handles audio input, and mtmd_init_from_file() loads a separate --mmproj GGUF projector model alongside the main model.
  tools/mtmd/mtmd.cpp#L582tools/mtmd/mtmd.cpp#L1085tools/mtmd/clip.cpp
• Model conversion is Python: convert_hf_to_gguf.py drives per-architecture converter classes in conversion/ registered via @ModelBase.register, writing GGUF through the gguf-py package (gguf-py/pyproject.toml, name=gguf v0.19.0); tensor name/layout tables live in gguf-py/gguf/constants.py.
  convert_hf_to_gguf.py#L15-25conversion/base.pygguf-py/pyproject.tomldocs/development/HOWTO-add-model.md
• The web UI is a SvelteKit + Vite + Tailwind app in tools/ui (package.json name=llama-ui) that is compiled to static assets and embedded into the server binary via a CMake-generated ui.cpp; LLAMA_USE_PREBUILT_UI downloads prebuilt assets from the HF bucket 'ggml-org/llama-ui' by default.
  tools/ui/package.jsontools/ui/CMakeLists.txt#L1-37tools/server/server-http.cpp#L5
• Top-level CMakeLists.txt is the single build system (cmake 3.14...3.28): version is LLAMA_VERSION 0.4.1-dev, feature options include LLAMA_BUILD_COMMON/TESTS/TOOLS/EXAMPLES/SERVER/APP/UI, LLAMA_OPENSSL, LLAMA_SUBPROCESS, LLAMA_LLGUIDANCE, plus deprecated-option shims mapping LLAMA_CUDA etc. to GGML_* options.
  CMakeLists.txt#L1-23CMakeLists.txt#L129-146CMakeLists.txt#L174-195

All 14 system map claims →

Feature catalog

User-facing surface splits into the `llama` unified binary subcommands, the llama-server HTTP API, the interactive `llama cli` chat client, offline analysis/conversion tools, and the embedded web UI. The server exposes OpenAI-compatible, Anthropic-compatible, Ollama-ish and legacy endpoints plus extras (rerank, infill, tokenize, slots save/load, LoRA hotswap, resumable streams, MCP tools). Feature scope for the server is deliberately bounded by tools/server/README-dev.md.

• llama-server registers a broad REST surface in llama_server(): /completion(s), /v1/completions, /chat/completions + /v1/chat/completions, /v1/responses, /v1/audio/transcriptions, /v1/messages + /v1/messages/count_tokens (Anthropic), /infill, /embedding(s) + /v1/embeddings, /rerank(ing) + /v1/rerank(ing), /tokenize, /detokenize, /apply-template, /chat/completions/input_tokens, /props, /models, /metrics, /health, /lora-adapters (GET/POST), /slots + /slots/:id_slot, and resumable-stream endpoints /v1/stream and /v1/streams/lookup.
  tools/server/server.cpp#L248-306
• Router mode activates when no model is specified: llama_server() rewires most route handlers to models_routes->proxy_post/proxy_get and adds /models/load, /models/unload, /models/sse and DELETE /models so models are loaded on demand into child processes.
  tools/server/server.cpp#L136-246tools/server/server-models.cpp#L255-367
• `llama cli` provides an interactive REPL with slash commands /exit, /regen, /clear, /read <file>, /glob <pattern>, /image, /audio, /video, a banner showing model/ftype/modalities, multimodal staging via base64 data URLs, and optional --out-file logging of the session.
  tools/cli/cli-context.cpp#L413-456tools/cli/cli-context.cpp#L517-555tools/cli/cli-context.cpp#L306-341
• Hidden CLI subcommands still ship in the binary: completion (text completion), bench, batched-bench, fit-params (auto-size params to fit device memory), quantize, and perplexity — all marked hidden=true in the cmds[] table.
  app/llama.cpp#L64-78tools/completion/completion.cpptools/llama-bench/llama-bench.cpptools/perplexity/perplexity.cpp
• Speculative decoding is a first-class feature via common/speculative.cpp: draft-model, MTP (multi-token prediction) and other types are parsed by common_speculative_types_from_names / inferred from GGUF metadata, and the server tracks draft-acceptance metrics per slot.
  common/speculative.h#L8-30common/speculative.cpptools/server/server-context.cpp#L4137-4139
• Structured output is supported through GBNF grammars (sample grammars in grammars/), json-schema-to-grammar.cpp converting JSON Schema to GBNF, and optional llguidance integration (LLAMA_LLGUIDANCE cmake option, llama_sampler_init_llg).
  common/json-schema-to-grammar.cppcommon/sampling.cpp#L215CMakeLists.txt#L146grammars/json.gbnf
• Chat formatting/parsing is handled by a dedicated Jinja engine (common/jinja, lexer+parser+runtime) plus a PEG-based parser framework (common/peg-parser.cpp, chat-peg-parser.cpp) and per-model tool-call parsers in common/parsers/ (kimi-k2, gpt-oss, ministral3, cohere2moe, etc.); common_chat_templates_init/apply produce the prompt and common_chat_parse extracts reasoning/tool calls.
  common/jinja/README.mdcommon/chat.h#L316-340common/parsers/kimi-k2.cppcommon/chat-peg-parser.cpp
• Model acquisition features: -hf <repo> Hugging Face downloads with an ETag-based file cache (hf-cache.cpp, write_etag/read_etag, atomic .tmp+rename writes), --model-url direct download, docker_repo support, and HF remote preset repos applied in router mode.
  common/download.cpp#L50-90common/hf-cache.cpptools/server/server.cpp#L122-134
• Server-side agent features: --tools enables built-in tools including exec_shell_command, file read/write tools, and MCP server integration (--mcp-*) where MCP servers run as stdio subprocesses via common_subproc; disabled features return a 403 'feature_disabled' response.
  tools/server/server-tools.cpp#L1239-1247tools/server/server-tools.cpp#L1960tools/server/server-mcp.cpp#L340-351tools/server/server.cpp#L312-371
• KV/memory management features: multi-slot continuous batching with n_parallel slots, unified KV cache (--kv-unified / --kv-unified-per-slot auto-sizing), prompt cache in RAM (--cache-ram / cache_idle_slots), context checkpoints (--ctx-checkpoints), context shifting, and slot save/restore endpoints.
  tools/server/server.cpp#L162-172tools/server/server-context.cpp#L1351-1370tools/server/server-context.cpp#L3625-3635
• LoRA adapters are loaded per request and hot-swappable: common_init_from_params loads params.lora_adapters via llama_adapter_lora_init, and the server exposes GET/POST /lora-adapters plus per-batch application via common_set_adapter_lora in update_slots().
  common/common.cpp#L1343-1358tools/server/server.cpp#L284-285tools/server/server-context.cpp#L2848-2865src/llama-adapter.cpp
• Quantization and analysis tools: llama-quantize (tools/quantize), llama-imatrix (importance matrix), llama-perplexity (PPL/KLD eval), llama-gguf-split, llama-tokenize, llama-tts, llama-export-lora, llama-cvector-generator, and rpc-server exposing a remote compute backend.
  tools/quantize/quantize.cpptools/imatrix/imatrix.cpptools/rpc/rpc-server.cpptools/gguf-splittools/tokenize

All 14 feature catalog claims →

Data flow

The canonical pipeline is: CLI args -> common_params (arg.cpp) -> model file resolution/download (download.cpp, hf-cache.cpp) -> GGUF parse + mmap via llama_model_loader -> per-arch llama_model subclass (tensors mapped into ggml backend buffers) -> llama_context with KV memory + graph builder -> llama_decode produces logits -> sampler chain produces tokens -> detokenize to text. In the server, HTTP JSON is tokenized into server_task objects flowing through server_queue -> server_slot -> a shared llama_batch -> llama_decode -> results streamed back as SSE.

• Argument handling flows through common_params_parser_init in common/arg.cpp, which builds a table of common_arg entries each supporting env-var fallback (LLAMA_ARG_* / LLAMA_ARG_NO_*), per-example gating (LLAMA_EXAMPLE_SERVER/CLI/MTMD/TTS), and preset-only flags.
  common/arg.cpp#L60-141common/arg.cpp#L3379-3505
• Model loading: common_init_from_params -> llama_model_load_from_file -> llama_model_load_from_file_impl -> llama_model_load constructs llama_model_loader (GGUF metadata + tensor index, mmap/mlock/direct-IO via params.load_mode), calls llama_model_create for arch dispatch, then load_hparams, load_vocab, load_stats, load_tensors in that order.
  common/common.cpp#L1329-1398src/llama.cpp#L316-378src/llama.cpp#L465-470
• GGUF input is parsed by ggml/src/gguf.cpp and wrapped by llama_model_loader which resolves multi-file splits (names like name-00002-of-00004.gguf) via llama_get_list_splits and provides typed metadata access through the GGUFMeta template layer.
  src/llama-model-loader.cpp#L80-120ggml/src/gguf.cpp
• Device placement is decided in llama_prepare_model_devices: it partitions ggml_backend_dev_t devices into RPC servers (front of list), discrete GPUs, and iGPUs (only if no dGPU), dedupes by device_id, honors LLAMA_SPLIT_MODE_TENSOR by wrapping devices in a ggml_backend_meta_device, and trims to main_gpu for LLAMA_SPLIT_MODE_NONE.
  src/llama.cpp#L158-313
• Context creation in common_init_from_params loads LoRA adapters, initializes per-sequence samplers (common_sampler_init, optionally passed to the context for backend sampling via cparams.samplers), then calls llama_init_from_model which builds KV memory (llama_memory_i implementations: kv-cache, kv-cache-iswa, memory-hybrid, memory-recurrent) and the graph pipeline.
  common/common.cpp#L1343-1408src/llama-memory.h#L73-129
• Tokenization flows through llm_tokenizer implementations in src/llama-vocab.cpp (SPM, BPE with regex pre-tokenizers, UGM, etc.) selected by vocab type from GGUF metadata; unicode handling lives in src/unicode.cpp/unicode-data.cpp.
  src/llama-vocab.cpp#L73-286
• Server request path: handle_completions_impl parses JSON, tokenizes via tokenize_input_prompts or process_mtmd_prompt (for multimodal), builds server_task objects with evaluated sampling params (server_schema::eval_llama_cmpl_schema), spawns child tasks when n_cmpl>1, and posts them to queue_tasks.
  tools/server/server-context.cpp#L4257-4342tools/server/server-task.cpp
• Slot scheduling: process_single_task handles COMPLETION/INFILL/EMBEDDING/RERANK by picking a slot via get_available_slot (deferring to queue_tasks.defer when none free or when child tasks lack slots) then launch_slot_with_task / launch_slots_with_parent_task.
  tools/server/server-context.cpp#L2375-2453tools/server/server-queue.h#L53-77
• Decode loop: update_slots() runs pre_decode (context shift, checkpoint creation) -> batch.render() -> chunked decode() calls llama_decode(ctx_tgt, batch_view) inside queue_tasks.yield_to_queue (so metrics tasks can run mid-decode) -> post_decode per chunk; llama_decode itself is async so timings only settle after llama_synchronize.
  tools/server/server-context.cpp#L2793-2907tools/server/server-context.cpp#L3647-3686
• Sampling is an ordered chain built by common_sampler_init: grammar/llg sampler first, then logit_bias, DRY, top-k, top-p, top-n-sigma, min-p, XTC, typical, temp_ext, infill, penalties, adaptive_p, and dist/temp at the end; chain order is data-driven from common_params_sampling.
  common/sampling.cpp#L187-404src/llama-sampler.cpp
• Results flow back through server_response queues: slots emit server_task_result_cmpl_partial/final which the HTTP layer formats as OAI SSE (format_oai_sse), Anthropic SSE (format_anthropic_sse), or OAI responses SSE, terminating with 'data: [DONE]'; non-stream requests collect via rd.wait_for_all.
  tools/server/server-context.cpp#L4348-4523tools/server/server-queue.h#L162-238
• CLI data flow: cli_context keeps an OpenAI-style messages array; user input (plus staged base64 media parts) is POSTed to /v1/chat/completions with stream=true and timings_per_token, and SSE deltas are rendered by ui::assistant_turn into reasoning/content streams.
  tools/cli/cli-context.cpp#L287-411

All 13 data flow claims →

Behavior traces

Three flows traced end-to-end: (1) `llama serve` startup including router vs single-model branches, (2) an HTTP chat completion through tokenize -> task queue -> slot -> decode -> SSE, and (3) `llama cli` startup which spawns an embedded server thread and streams chat over loopback HTTP.

• Server startup trace: app main() matches 'serve' -> llama_server(argc,argv) -> setlocale/SIGPIPE ignore -> common_init() -> server_stream_session_manager_start() -> common_params_parse(LLAMA_EXAMPLE_SERVER) -> llama_backend_init() + llama_numa_init() -> llama_server(params,...) which resolves HF presets, sizes KV pool, and constructs server_mcp, server_context, server_http_context.
  app/llama.cpp#L129-143tools/server/server.cpp#L88-190
• Server startup (continued): ctx_http.init(params) builds the httplib (SSL)Server -> routes registered on ctx_http with ex_wrapper -> mcp_mgr.start + tools.setup -> model download via common_models_handler_apply -> ctx_http.start() (before model load so /health works) -> ctx_server.load_model(params) -> routes.update_meta -> ctx_server.start_loop() blocks the main thread in queue_tasks.start_loop().
  tools/server/server.cpp#L192-246tools/server/server.cpp#L383-493tools/server/server.cpp#L540-558
• Inside load_model -> init(): wires queue callbacks (on_new_task -> process_single_task, on_update_slots -> update_slots, on_sleeping_state -> handle_sleeping_state), inits metrics, reads LLAMA_TRACE / LLAMA_SERVER_SLOTS_DEBUG / LLAMA_SERVER_SLOTS_N_DIFF env vars, sizes the shared llama_batch to max(n_batch, n_parallel), creates the RAM prompt cache, and initializes chat templates via common_chat_templates_init.
  tools/server/server-context.cpp#L1290-1460
• Chat completion trace: POST /v1/chat/completions -> routes.post_chat_completions -> handle_completions_impl -> tokenize/process_mtmd_prompt -> server_task(type=COMPLETION) per input -> rd.post_tasks -> server_queue::start_loop drains queue -> process_single_task -> get_available_slot -> launch_slot_with_task -> next update_slots() iteration batches it.
  tools/server/server-context.cpp#L4257-4342tools/server/server-queue.cpp#L278-316tools/server/server-context.cpp#L2382-2453
• Generation loop trace: update_slots() early-returns when all slots idle (else self-posts SERVER_TASK_TYPE_NEXT_RESPONSE) -> pre_decode() applies context-shift/checkpoints -> batch.render() -> for each n_batch chunk decode() -> queue_tasks.yield_to_queue wrapping llama_decode(ctx_tgt, batch_view) + llama_synchronize -> post_decode() samples and emits partial/final results.
  tools/server/server-context.cpp#L2807-2907tools/server/server-context.cpp#L3673-3686
• Streaming response trace: first rd.next() result decides headers; then res->set_next drives chunked output — format_oai_sse/format_anthropic_sse/format_oai_resp_sse per result, SSE keepalive ping ':' when sse_ping_interval elapses, 'data: [DONE]' terminator for OAI formats; errors mid-stream are formatted as SSE error events.
  tools/server/server-context.cpp#L4380-4523
• CLI startup trace: llama_cli -> common_init() -> common_params_parse(LLAMA_EXAMPLE_CLI) -> install SIGINT/SIGTERM handler -> cli_context(params) -> init(): either connect to --server-base or cli_server.start() spawns llama_server() on a std::thread with a random port, wait_ready() polls GET /health until 200, then client.wait_health and list_and_ask_models() (GET /v1/models) pick a model.
  tools/cli/cli.cpp#L33-64tools/cli/cli-server.h#L34-84tools/cli/cli-context.cpp#L94-183
• CLI turn trace: run() prints banner -> user_turn.read_input() -> slash-command dispatch (/regen pops last message, /clear resets, /image|/audio|/video stages base64, /read and /glob inline file text) -> push_user_message -> generate_completion POSTs SSE to /v1/chat/completions -> delta.reasoning_content/delta.content rendered via ui::assistant_turn, timings pulled from chunk.timings.
  tools/cli/cli-context.cpp#L413-560tools/cli/cli-context.cpp#L350-411
• Model load trace (persistence path): llama_model_load_from_file -> llama_model_load_from_file_impl validates exactly one of metadata/path/file, installs a default progress callback -> llama_model_load -> llama_model_loader ctor (gguf parse, split resolution, mmap) -> llama_model_create(ml, params) -> llama_prepare_model_devices -> model->load_hparams -> load_vocab -> load_stats -> print_info -> load_tensors (supports vocab_only early return and -2 cancellation via progress callback).
  src/llama.cpp#L316-444src/llama-model-loader.cpp#L80-105
• Router child spawn trace: server_models::load (server-models.cpp) builds child argv, sets subprocess_option_no_window|combined_stdout_stderr, calls inst.subproc->sproc.create(child_args, options, child_env); a single waiter thread watches all children, parses 'cmd_child_to_router:state:' lines from stdout and calls on_child_exit on EOF/exit; request_child_exit writes 'cmd_router_to_child:exit' to the child's stdin.
  tools/server/server-models.cpp#L45-80tools/server/server-models.cpp#L184-255tools/server/server-models.cpp#L1163-1280
• Shutdown trace: SIGINT -> signal_handler -> shutdown_handler -> (single-model) mcp_mgr.shutdown() + ctx_server.terminate() which unblocks start_loop -> clean_up(): server_stream_session_manager_stop, ctx_http.stop, ctx_server.terminate, llama_backend_free; a second Ctrl+C force-exits with exit(1).
  tools/server/server.cpp#L25-49tools/server/server.cpp#L448-558

Testing signals

C++ tests live in tests/ and register with CTest via llama_build_and_test/llama_test helpers; coverage spans tokenizers (against models/ggml-vocab-*.gguf fixtures), grammars, chat templates/parsers, KV/recurrent state, samplers, quantization, and backend ops. The server has a parallel pytest suite (tools/server/tests) that spawns real llama-server processes, and the web UI has Vitest (unit/client/ui projects) plus Playwright e2e. CI runs ctest -L main and dedicated per-backend workflows.

• Test registration is CMake-driven: tests/CMakeLists.txt defines llama_build, llama_test, llama_test_cmd, llama_build_and_test helpers; most tests link llama-common (some also libllama) and are labeled 'main' or 'model' for ctest -L filtering.
  tests/CMakeLists.txt#L3-117
• Tokenizer coverage is fixture-heavy: test-tokenizer-0 is built once and instantiated ~15 times against models/ggml-vocab-*.gguf files (bert-bge, command-r, deepseek-coder, gemma-4, gpt-2, llama-bpe/spm, qwen2, qwen35, ...); several slower tokenizer tests are commented out.
  tests/CMakeLists.txt#L120-191models/
• Several tests are gated: Windows+shared-libs excludes tests using non-exported internals (test-unicode, test-sampling, test-grammar-*, test-chat, ...); test-opt is skipped under ASAN or GGML_SCHED_NO_REALLOC; backend-direct tests (test-barrier, test-quantize-*, test-rope) need non-DL CPU backend.
  tests/CMakeLists.txt#L154-172tests/CMakeLists.txt#L304-332
• A dummy-model fixture system exists: test-llama-archs generates tiny GGUFs for every architecture (test-generate-models ctest fixture), then test-recurrent-state-rollback and test-save-load-state run against generated qwen35-dense, nemotron_h-dense, deepseek4-moe, kimi-k3-moe models.
  tests/CMakeLists.txt#L195-256tests/test-llama-archs.cpp
• A real model is downloaded in CI: test-download-model fetches tinyllamas/stories15M-q4_0.gguf (SHA256-pinned) via cmake/download-models.cmake as a ctest fixture required by test-thread-safety and test-state-restore-fragmented.
  tests/CMakeLists.txt#L279-319
• Chat/template stack has dedicated tests: test-chat (links server-context lib), test-chat-template, test-chat-peg-parser, test-chat-auto-parser, test-jinja (plus a -py labeled variant), test-json-schema(-to-grammar), test-peg-parser with a suite of peg-parser/test-*.cpp files; test-chat-analysis is built but deliberately not registered (manual debug tool).
  tests/CMakeLists.txt#L163-166tests/CMakeLists.txt#L258-277
• Server has a pytest suite in tools/server/tests: tests.sh runs pytest -n auto --dist=worksteal with a 'slow' marker; utils.py ServerProcess spawns the real llama-server binary on a port with a tiny stories260K model; 27 unit files cover chat completion, OAI/Anthropic/GCP compat, embeddings, rerank, infill, lora, mcp, router, security, sleep, slots, speculative, streaming, vision, tool calls.
  tools/server/tests/tests.shtools/server/tests/utils.py#L38-60tools/server/tests/unit/test_security.py
• Security tests exist for API keys: test_security.py asserts /health is public, static UI assets bypass auth, wrong/missing Bearer keys get 401 authentication_error, and a valid key returns 200.
  tools/server/tests/unit/test_security.py
• UI testing is three-tiered: vitest projects 'unit', 'client' (browser mode via playwright provider) and 'ui', plus 'test:e2e' playwright tests and storybook; scripts in tools/ui/package.json wire them into npm test.
  tools/ui/package.json#L17-24tools/ui/vite.config.tstools/ui/playwright.config.ts
• CI coverage: .github/workflows/server.yml runs the server pytest suite on PRs touching C/C++ or tools/server; build-cpu.yml runs 'ctest -L main --verbose --timeout 900'; separate workflows cover CUDA/Metal/Vulkan/SYCL/sanitizers/server-sanitize; python-lint and python-type-check gate the conversion code.
  .github/workflows/server.yml#L1-40.github/workflows/build-cpu.yml#L127.github/workflows/server-sanitize.yml.github/workflows/python-lint.yml
• Coverage gaps: test-backend-ops and test-tokenizer-1-bpe exist as build targets but have most of their test registrations commented out ('disabled due to slowness'); test-double-float is commented as SLOW; ggml backend compute correctness relies heavily on test-backend-ops which is built but only selectively registered.
  tests/CMakeLists.txt#L177-193tests/CMakeLists.txt#L310

Risk map

Biggest risks: server-side command execution features (exec_shell_command, MCP stdio subprocesses, router child spawning, CORS proxy) which are opt-in but powerful; unauthenticated-by-default HTTP API; remote content trust (HF downloads, remote presets, prebuilt UI assets); and a very large single-threaded decode loop in server-context.cpp (~5.5k lines) where a fatal error aborts all slots. GGML_ABORT/GGML_ASSERT turn many error paths into crashes rather than graceful failures.

• exec_shell_command is a built-in server tool that runs arbitrary shell commands via run_subprocess (sheredom subprocess.h, subprocess_option_search_user_path|inherit_environment); an 'isolate' mode can redirect file/exec ops through a container or remote host transport — gated behind --tools/--tools-runtime which the server logs as experimental security warnings.
  tools/server/server-tools.cpp#L187-207tools/server/server-tools.cpp#L516-624tools/server/server-tools.cpp#L1239-1247tools/server/server.cpp#L350-380
• MCP integration spawns arbitrary configured commands as stdio subprocesses (server_mcp_stdio wraps common_subproc) with ndjson JSON-RPC over stdin/stdout; MCP server configs can be supplied as JSON including Cursor 'mcpServers' format.
  tools/server/server-mcp.cpp#L340-351tools/server/server-mcp.cpp#L132-176common/subproc.cpp#L30-49
• Router mode spawns child llama-server processes with attacker-influenceable args (model params come from HTTP /models/load requests); it requires LLAMA_SUBPROCESS and throws when subprocess support is compiled out.
  tools/server/server-models.cpp#L1163-1189tools/server/server-models.cpp#L1923-1924CMakeLists.txt#L104-110
• Auth is optional: API keys come from --api-key/--api-key-file (LLAMA_ARG_API_KEY env); the server only warns (not fails) when cors_origins='*' with no keys, and prints 'do not expose to untrusted environments' warnings for router/tools/MCP features.
  common/arg.cpp#L3484-3505tools/server/server.cpp#L324-341tools/server/server.cpp#L373-380
• A /cors-proxy endpoint proxies arbitrary GET/POST requests when --ui-mcp-proxy is enabled (experimental, used by the Web UI for MCP), effectively an open proxy surface on the server port.
  tools/server/server.cpp#L328-341tools/server/server-cors-proxy.h
• Remote model handling trusts network content: -hf downloads GGUFs and preset .ini files from HF (with an explicit 'only use presets that you can trust' warning), and LLAMA_USE_PREBUILT_UI pulls UI assets from a HF bucket at build time.
  tools/server/server.cpp#L520-524common/download.cpptools/ui/CMakeLists.txt#L1-37
• `llama update` shells out: on LLAMA_INSTALL_BUILD it runs system('curl ... | sh') or powershell irm|iex against https://llama.app — a code-execution path dependent on TLS and remote server integrity.
  app/llama.cpp#L27-41
• Error handling frequently escalates to process abort: GGML_ABORT/GGML_ASSERT appear on hot paths (e.g. >3 consecutive empty decode batches aborts; model not implementing llama_model_base aborts), so malformed inputs or edge states can take down the whole server rather than one request.
  tools/server/server-context.cpp#L3652-3662src/llama.cpp#L332-335tools/server/server-context.cpp#L4395-4398
• Decode failure blast radius: when llama_decode returns nonzero, decode() sends errors to ALL processing slots and clears the entire context per-slot ('it's complicated to keep track of how much of the current batch has been processed'), so one bad request can kill concurrent sessions.
  tools/server/server-context.cpp#L3688-3719
• server-context.cpp is a 5,558-line pimpl monolith owning slots, batching, checkpointing, prompt cache, metrics, LoRA, speculative decode and all route handlers — high coupling makes regressions likely; code comments admit 'alora handling is too messy, need to refactor'.
  tools/server/server-context.cpp#L2853tools/server/server-context.cpp
• GGUF/mmap input parsing is a trust boundary: llama_model_loader and gguf.cpp parse arbitrary model files (metadata strings, tensor offsets) and mmap them directly; params.check_tensors exists but untrusted model files remain a memory-safety attack surface. (inferred)
  src/llama-model-loader.cpp#L1-30src/llama-mmap.cppggml/src/gguf.cpp
• SSL is optional and off unless CPPHTTPLIB_OPENSSL_SUPPORT is compiled in AND both --ssl-key-file/--ssl-cert-file are given; otherwise the API (including Bearer keys) runs plaintext HTTP — default port :8080 with a pending change to :9931.
  tools/server/server-http.cpp#L107-124CMakeLists.txt#L144tools/server/server.cpp#L514-518

All 13 risk map claims →

Extension points

Primary extension seams: new model architectures (conversion/ ModelBase.register + src/models/*.cpp + llama_model_mapping switch + gguf constants), new ggml backends (ggml_backend_reg_t + dlopen registry), new samplers (llama_sampler interface + common_sampler_init chain), new chat/tool parsers (common/parsers + PEG/jinja), new CLI args (common_arg table), new server endpoints (ctx_http.get/post + server_routes handlers), new built-in tools (server_tool subclasses), and new unified-binary subcommands (cmds[] table).

• New model architecture recipe is documented: register a TextModel/MmprojModel subclass via @ModelBase.register in conversion/, add MODEL_ARCH + MODEL_TENSORS entries in gguf-py/gguf/constants.py, add LLM_ARCH_* enum, a src/models/<arch>.cpp graph builder, and a case in llama_model_mapping — see docs/development/HOWTO-add-model.md.
  docs/development/HOWTO-add-model.mdsrc/llama-model.cpp#L43-373gguf-py/gguf/constants.pyconversion/base.py
• ggml backends are plugins: a backend exports a ggml_backend_*_reg() registered statically under #ifdef GGML_USE_<X> or loaded at runtime by ggml_backend_load(path) which dlopens a shared library — GGML_BACKEND_DL builds each backend as a separate .so.
  ggml/src/ggml-backend-reg.cpp#L115-200ggml/src/ggml-backend-reg.cpp#L393ggml/src/ggml-backend-dl.cpp
• Samplers implement the llama_sampler interface (src/llama-sampler.cpp provides ~20 stock samplers); common_sampler_init assembles them into a chain in a data-driven order, so adding a sampler means a new llama_sampler_init_* plus a slot in the ordering switch.
  src/llama-sampler.cppcommon/sampling.cpp#L187-404
• Chat/tool-call parsing is extensible at three layers: the standalone jinja engine (common/jinja) for templates, per-model PEG parsers in common/parsers/*.cpp (one file per model family), and chat-auto-parser which auto-detects model-specific features — all surfaced via common_chat_templates_apply/common_chat_parse.
  common/chat.h#L316-340common/parsers/kimi-k2.cppcommon/chat-auto-parser.hcommon/jinja/README.md
• CLI arguments are declarative: each option is a common_arg entry in common_params_parser_init with flags, env var (set_env), applicable examples (set_examples/set_excludes), and handler lambda; get_common_arg_defs exposes the table for introspection.
  common/arg.cpp#L77-141common/arg.cpp#L3379-3420
• Server endpoints plug in via ctx_http.get/post/del + a handler on server_routes (routes registered in llama_server); all handlers are wrapped by ex_wrapper which maps std::invalid_argument to 400 and other exceptions to 500 with an OAI-style error body.
  tools/server/server.cpp#L54-86tools/server/server.cpp#L248-306
• Built-in server tools extend the server_tool interface (name/schema/run) and are appended in the tools registry (tools.push_back(std::make_unique<server_tool_...>())); an isolate-backed tools_io abstracts file/exec ops for sandboxed runtimes.
  tools/server/server-tools.cpp#L149-207tools/server/server-tools.cpp#L516-624tools/server/server-tools.cpp#L1960
• New unified-binary subcommands: add an entry to the cmds[] table in app/llama.cpp (name, desc, aliases, hidden flag, func) — hidden commands exist but aren't listed in default help.
  app/llama.cpp#L49-78
• KV/memory is abstracted behind llama_memory_i (seq_rm/seq_cp/seq_keep/get_can_shift/apply); multiple implementations already coexist (llama-kv-cache, -iswa, -dsa, -dsv4, -msa, memory-hybrid, memory-recurrent) selected per architecture.
  src/llama-memory.h#L73-129src/llama-kv-cache.cppsrc/llama-memory-recurrent.cpp
• Structured output plugs through grammars: standalone .gbnf files in grammars/, JSON-Schema->GBNF conversion, and an optional llguidance backend (LLAMA_LLGUIDANCE) via llama_sampler_init_llg in the sampler chain.
  grammars/README.mdcommon/json-schema-to-grammar.cppcommon/sampling.cpp#L215
• Speculative decoding types are enumerable and extensible: common_speculative_type_from_name / _from_gguf map names and GGUF metadata to enum common_speculative_type (draft, MTP, ...), implemented in common/speculative.cpp (~3k lines).
  common/speculative.h#L8-40common/speculative.cpp
• MCP servers extend tool surface without code changes: server_mcp_server_config::parse_from_json accepts Cursor-style 'mcpServers' JSON to spawn stdio MCP servers whose tools are listed via list_tools and invoked via call_tool.
  tools/server/server-mcp.cpp#L132-176tools/server/server-mcp.cpp#L283-334

Contributor handoff

Read AGENTS.md first — it is strict: ASCII-only, concise comments, no new test files without maintainer approval, never commit/push/PR on the user's behalf, reuse existing infra. Build with cmake -B build && cmake --build build -j; run C++ tests via ctest and server tests via tools/server/tests/tests.sh. Key files to read before editing: common/arg.cpp (params), tools/server/README-dev.md (server feature scope), docs/development/HOWTO-add-model.md (arch additions), and the pimpl of whatever you touch (server_context_impl, llama_model subclasses).

• AGENTS.md is mandatory and unusual: it forbids agents from writing PR descriptions/comments/reviewer responses, forbids git push/gh pr create entirely (project ban risk), requires 'Assisted-by:' not 'Co-authored-by:' in human-approved commits, and tells fully autonomous agents to stop and exclude llama.cpp from contribution targets.
  AGENTS.md
• Style rules enforced by maintainers: no emdashes/unicode arrows (use ASCII '-', '->', 'x', '...'), comments are 1-2 lines max in ASD-STE100 simplified English, no hard-wrapping comments, no comment-first-then-code, and do NOT split a line mid-sentence to fit a column width.
  AGENTS.md
• Do not add new files under tests/ without maintainer approval — reuse existing test infrastructure; test-chat links against the server-context library so server chat changes can be exercised from tests/test-chat.cpp.
  AGENTS.mdtests/CMakeLists.txt#L163-166
• Server feature scope is contractually bounded by tools/server/README-dev.md: in-scope = inference, chat/tool calling, API compat, multimodal, memory mgmt, Web UI needs; out-of-scope = server-side agentic loops, internal-state APIs, model-specific features; file-IO features (MCP, model save/load) must be disabled by default for security.
  tools/server/README-dev.md
• Build: cmake -B build (Release default, BUILD_SHARED_LIBS on) then cmake --build build -j produces bin/llama plus llama-* tools; LLAMA_BUILD_UI embeds the web UI; LLAMA_SUBPROCESS is off on iOS/Android/Emscripten which disables server router mode and server tools.
  CMakeLists.txt#L104-146tools/server/server-models.cpp#L1923-1924
• Test invocation: ctest --test-dir build (label 'main' runs the core suite; 'model' needs downloaded models); server suite: cd tools/server/tests && ./tests.sh (needs PYTEST deps, spawns built server); UI: cd tools/ui && npm test (vitest+playwright).
  tests/CMakeLists.txt#L26-47tools/server/tests/tests.shtools/ui/package.json#L17-24
• Before touching CLI behavior read common/arg.cpp's common_params_parser_init — every flag is a common_arg with examples/env/handler; adding a flag means one table entry plus documentation regeneration; LLAMA_EXAMPLE_* gating controls which binaries accept the flag.
  common/arg.cpp#L77-141common/arg.h
• Server edits: the inference core is server_context_impl inside tools/server/server-context.cpp (pimpl over server_context); slot scheduling lives in process_single_task/get_available_slot/launch_slot_with_task; decode loop is update_slots -> pre_decode/decode/post_decode; HTTP routes are registered in server.cpp not in server-context.
  tools/server/server-context.cpp#L2375-2453tools/server/server-context.cpp#L2793-2907tools/server/server.cpp#L248-306
• Adding a model arch requires coordinated edits across five places: conversion/<arch>.py (ModelBase.register), gguf-py/gguf/constants.py (MODEL_ARCH/MODEL_TENSORS), src/llama-arch.* (LLM_ARCH_*), src/models/<arch>.cpp (graph builder), and llama_model_mapping in src/llama-model.cpp — HOWTO-add-model.md is the canonical guide.
  docs/development/HOWTO-add-model.mdsrc/llama-model.cpp#L43-373src/llama-arch.h
• The chat stack has traps: llama.cpp uses its own Jinja engine in common/jinja (NOT minja — AGENTS.md explicitly warns about stale knowledge); tool-call parsers are PEG-based in common/peg-parser.cpp + common/parsers; use test-chat-peg-parser/test-jinja to validate changes.
  AGENTS.mdcommon/jinja/README.mdtests/CMakeLists.txt#L258-263
• Concurrency model: a single decode thread runs queue_tasks.start_loop -> update_slots; HTTP handlers run on httplib threads and only communicate via queue_tasks.post + queue_results; decode yields via yield_to_queue only for metrics/slot-read tasks — never call llama_decode off the main loop.
  tools/server/server-queue.h#L14-45tools/server/server-context.cpp#L2375-2380tools/server/server-queue.cpp#L278-316
• Known foot-guns: GGML_ABORT crashes on error paths; llama_decode errors clear entire slots; llama-server.cpp default port 8080 is slated to change to 9931; embeddings require n_batch<=n_ubatch (auto-clamped with warning); 'completion' is the legacy hidden tool — prefer server/cli paths for new work.
  tools/server/server-context.cpp#L3652-3719tools/server/server.cpp#L148-152tools/server/server.cpp#L514-518app/llama.cpp#L69

Languages

Start reading here

Probable source entrypoint: src/ — inferred from the directory tree and manifest, not the upstream docs.

Source roots

src/ (223 files) app/ (3 files)

Repository structure

Docs found: AGENTS.mdCLAUDE.mdCONTRIBUTING.mdREADME.mddocs/android.mddocs/autoparser.mddocs/backend/BLIS.mddocs/backend/CANN.mddocs/backend/CUDA-FEDORA.mddocs/backend/ET.md

Package

pyproject.toml — pytorch v0.0.0, 8 declared dependencies.

torch{{{]pytest##

CI & config entrypoints

• README.md — Repository readme
• AGENTS.md — Agent instructions
• CLAUDE.md — Claude instructions
• tools/ui/playwright.config.ts — Playwright e2e config
• tools/ui/vite.config.ts — Vite config
• .github/workflows/gguf-publish.yml — GitHub Actions workflow
• .github/workflows/build-cann.yml — GitHub Actions workflow
• .github/workflows/fusion.yml — GitHub Actions workflow
• .github/workflows/editorconfig.yml — GitHub Actions workflow
• .github/workflows/ui-build-self-hosted.yml — GitHub Actions workflow

Structural health signals

Average score 6.5/10 across 96 analyzed source files; 50 hotspots surfaced.

• scripts/compare-llama-bench.py — 1102 lines, score 4.7 (Large file, Deep nesting)
• tools/ui/src/lib/vendors/nerdamer-prime/nerdamer.core.js — 2860 lines, score 5 (Large file, Deep nesting)
• tools/ui/src/lib/vendors/nerdamer-prime/Algebra.js — 2174 lines, score 5 (Large file, Deep nesting)

These are deterministic review leads — size, churn, and structural signals — not proof of a bug in this project.

Recent history

• 7ab4ee7baad2 2026-09-22 — chat : Fix Muse Glimmer tool-call first parser error (#29242)
• 0ee9435b8f91 2026-09-22 — ci : publish snapdragon builds in release workflow (#29007)
• 8cfc315a8abd 2026-09-22 — Add close button to UI toasts (#28246)
• ec5a12b85ae3 2026-09-21 — opencl: add A8 Q4_0 non-MoE dp4a binary kernel (#29055)
• c550d2f60bde 2026-09-22 — ci : update Level Zero SDK to v1.33.1 and enable the L0/oneDNN CMake flags in the SYCL job (#29230)