ggml-org/llama.cpp

User-facing surface splits into the `llama` unified binary subcommands, the llama-server HTTP API, the interactive `llama cli` chat client, offline analysis/conversion tools, and the embedded web UI. The server exposes OpenAI-compatible, Anthropic-compatible, Ollama-ish and legacy endpoints plus extras (rerank, infill, tokenize, slots save/load, LoRA hotswap, resumable streams, MCP tools). Feature scope for the server is deliberately bounded by tools/server/README-dev.md.

• llama-server registers a broad REST surface in llama_server(): /completion(s), /v1/completions, /chat/completions + /v1/chat/completions, /v1/responses, /v1/audio/transcriptions, /v1/messages + /v1/messages/count_tokens (Anthropic), /infill, /embedding(s) + /v1/embeddings, /rerank(ing) + /v1/rerank(ing), /tokenize, /detokenize, /apply-template, /chat/completions/input_tokens, /props, /models, /metrics, /health, /lora-adapters (GET/POST), /slots + /slots/:id_slot, and resumable-stream endpoints /v1/stream and /v1/streams/lookup.
  tools/server/server.cpp#L248-306
• Router mode activates when no model is specified: llama_server() rewires most route handlers to models_routes->proxy_post/proxy_get and adds /models/load, /models/unload, /models/sse and DELETE /models so models are loaded on demand into child processes.
  tools/server/server.cpp#L136-246tools/server/server-models.cpp#L255-367
• `llama cli` provides an interactive REPL with slash commands /exit, /regen, /clear, /read <file>, /glob <pattern>, /image, /audio, /video, a banner showing model/ftype/modalities, multimodal staging via base64 data URLs, and optional --out-file logging of the session.
  tools/cli/cli-context.cpp#L413-456tools/cli/cli-context.cpp#L517-555tools/cli/cli-context.cpp#L306-341
• Hidden CLI subcommands still ship in the binary: completion (text completion), bench, batched-bench, fit-params (auto-size params to fit device memory), quantize, and perplexity — all marked hidden=true in the cmds[] table.
  app/llama.cpp#L64-78tools/completion/completion.cpptools/llama-bench/llama-bench.cpptools/perplexity/perplexity.cpp
• Speculative decoding is a first-class feature via common/speculative.cpp: draft-model, MTP (multi-token prediction) and other types are parsed by common_speculative_types_from_names / inferred from GGUF metadata, and the server tracks draft-acceptance metrics per slot.
  common/speculative.h#L8-30common/speculative.cpptools/server/server-context.cpp#L4137-4139
• Structured output is supported through GBNF grammars (sample grammars in grammars/), json-schema-to-grammar.cpp converting JSON Schema to GBNF, and optional llguidance integration (LLAMA_LLGUIDANCE cmake option, llama_sampler_init_llg).
  common/json-schema-to-grammar.cppcommon/sampling.cpp#L215CMakeLists.txt#L146grammars/json.gbnf
• Chat formatting/parsing is handled by a dedicated Jinja engine (common/jinja, lexer+parser+runtime) plus a PEG-based parser framework (common/peg-parser.cpp, chat-peg-parser.cpp) and per-model tool-call parsers in common/parsers/ (kimi-k2, gpt-oss, ministral3, cohere2moe, etc.); common_chat_templates_init/apply produce the prompt and common_chat_parse extracts reasoning/tool calls.
  common/jinja/README.mdcommon/chat.h#L316-340common/parsers/kimi-k2.cppcommon/chat-peg-parser.cpp
• Model acquisition features: -hf <repo> Hugging Face downloads with an ETag-based file cache (hf-cache.cpp, write_etag/read_etag, atomic .tmp+rename writes), --model-url direct download, docker_repo support, and HF remote preset repos applied in router mode.
  common/download.cpp#L50-90common/hf-cache.cpptools/server/server.cpp#L122-134
• Server-side agent features: --tools enables built-in tools including exec_shell_command, file read/write tools, and MCP server integration (--mcp-*) where MCP servers run as stdio subprocesses via common_subproc; disabled features return a 403 'feature_disabled' response.
  tools/server/server-tools.cpp#L1239-1247tools/server/server-tools.cpp#L1960tools/server/server-mcp.cpp#L340-351tools/server/server.cpp#L312-371
• KV/memory management features: multi-slot continuous batching with n_parallel slots, unified KV cache (--kv-unified / --kv-unified-per-slot auto-sizing), prompt cache in RAM (--cache-ram / cache_idle_slots), context checkpoints (--ctx-checkpoints), context shifting, and slot save/restore endpoints.
  tools/server/server.cpp#L162-172tools/server/server-context.cpp#L1351-1370tools/server/server-context.cpp#L3625-3635
• LoRA adapters are loaded per request and hot-swappable: common_init_from_params loads params.lora_adapters via llama_adapter_lora_init, and the server exposes GET/POST /lora-adapters plus per-batch application via common_set_adapter_lora in update_slots().
  common/common.cpp#L1343-1358tools/server/server.cpp#L284-285tools/server/server-context.cpp#L2848-2865src/llama-adapter.cpp
• Quantization and analysis tools: llama-quantize (tools/quantize), llama-imatrix (importance matrix), llama-perplexity (PPL/KLD eval), llama-gguf-split, llama-tokenize, llama-tts, llama-export-lora, llama-cvector-generator, and rpc-server exposing a remote compute backend.
  tools/quantize/quantize.cpptools/imatrix/imatrix.cpptools/rpc/rpc-server.cpptools/gguf-splittools/tokenize
• Google Cloud Vertex AI compat: gcp_params in server-http.cpp reads AIP_MODE/AIP_HEALTH_ROUTE/AIP_PREDICT_ROUTE/AIP_HTTP_PORT env vars and remaps health/predict routes and port when AIP_MODE=PREDICTION.
  tools/server/server-http.cpp#L60-103tools/server/server.cpp#L308-309
• The web UI is a full chat frontend (SvelteKit static build embedded in the binary) with model management, MCP support, PWA assets, storybook stories, and its own Playwright e2e + Vitest suites configured in tools/ui/package.json.
  tools/ui/package.json#L6-27tools/ui/vite.config.tstools/ui/playwright.config.ts