ggml-org/llama.cpp

The repo is a CMake C/C++ monorepo layered as: ggml/ (tensor ops + hardware backend plugins, statically registered or dlopen'd), src/ (libllama: GGUF model loading, vocab/tokenizers, KV-cache memory, compute graph builders per architecture, samplers, public C API in include/llama.h), common/ (llama-common utility lib: arg parsing, chat templates via a custom Jinja engine, sampling orchestration, HF downloading, subprocess helpers), tools/ (user-facing binaries incl. server, cli, mtmd multimodal, quantize, perplexity, bench), app/ (unified `llama` binary), and Python conversion tooling (conversion/, gguf-py/, convert_hf_to_gguf.py). Tests live in tests/ (CTest C++ binaries) and tools/server/tests (pytest). The web UI is a SvelteKit app in tools/ui embedded into the server binary at build time.

• The unified `llama` binary is a thin dispatcher: app/llama.cpp defines a static command table `cmds[]` mapping subcommands (serve, cli, update, download, completion, bench, batched-bench, fit-params, quantize, perplexity, version, licenses, help) to entry functions like llama_server and llama_cli, and sets the LLAMA_APP_CMD env var before invoking them so router child processes re-invoke correctly.
  app/llama.cpp#L64-78app/llama.cpp#L129-143
• The public API is a C ABI in include/llama.h (~1646 lines, ~246 LLAMA_API declarations) implemented primarily by src/llama.cpp, which forwards into internal classes llama_model, llama_context, llama_vocab, and llama_memory.
  include/llama.hsrc/llama.cpp#L1-30
• Model instantiation is a giant architecture switch: llama_model_mapping() in src/llama-model.cpp maps each LLM_ARCH_* enum to a `new llama_model_<arch>(params)` subclass (e.g. llama_model_llama, llama_model_deepseek2); unknown archs throw 'unsupported model architecture'.
  src/llama-model.cpp#L43-373
• Per-architecture graph builders live in src/models/ (156 files, e.g. llama.cpp, deepseek2.cpp, qwen3.cpp) plugged into llama-graph.cpp which assembles ggml compute graphs executed via llama_context::decode().
  src/llama-graph.cppsrc/llama-context.cpp#L4326-4335src/llama-graph.h#L101-363
• ggml is vendored in ggml/ with a backend registry: ggml_backend_registry's constructor statically registers compile-time backends (CUDA, Metal, SYCL, Vulkan, WebGPU, zDNN, VirtGPU, OpenCL, ZenDNN, Hexagon, CANN, BLAS, RPC, OpenVINO, ET, CPU) and ggml_backend_load()/ggml_backend_load_all() can dlopen backend shared libraries at runtime (GGML_BACKEND_DL).
  ggml/src/ggml-backend-reg.cpp#L115-174ggml/src/ggml-backend-reg.cpp#L393ggml/src/ggml-backend-reg.cpp#L574-603
• common/ builds the llama-common static library holding shared tool infrastructure: common_params + arg parsing (arg.cpp, ~4.7k lines), chat template/tool-call handling (chat.cpp + jinja/ + parsers/), sampler orchestration (sampling.cpp), speculative decoding (speculative.cpp), HF/URL downloading (download.cpp, hf-cache.cpp), subprocess wrapper (subproc.cpp) and a cpp-httplib HTTP client helper (http.h).
  common/arg.cpp#L77-84common/chat.h#L316-340common/sampling.cpp#L187-210common/download.cpp#L45-90common/http.h#L100-118
• tools/server implements the OpenAI/Anthropic-compatible HTTP server on cpp-httplib (vendor/cpp-httplib): server-http.cpp wraps httplib::Server/SSLServer, server.cpp registers all routes, server-context.cpp owns the inference slots/queue, and server-models.cpp implements router mode by spawning child llama-server subprocesses.
  tools/server/server-http.cpp#L7-22tools/server/server.cpp#L241-306tools/server/server-models.cpp#L1163-1189vendor/cpp-httplib
• tools/cli is an interactive chat client that embeds the server: cli_context::init() spawns llama_server() on a std::thread bound to a random free port (cli_server::start) and all generation goes through HTTP SSE to /v1/chat/completions rather than calling libllama directly.
  tools/cli/cli-server.h#L33-56tools/cli/cli-context.cpp#L94-151tools/cli/cli-context.cpp#L350-411
• Multimodal support is a separate mtmd library in tools/mtmd: clip.cpp (~6k lines) implements the vision encoder stack, mtmd-audio.cpp handles audio input, and mtmd_init_from_file() loads a separate --mmproj GGUF projector model alongside the main model.
  tools/mtmd/mtmd.cpp#L582tools/mtmd/mtmd.cpp#L1085tools/mtmd/clip.cpp
• Model conversion is Python: convert_hf_to_gguf.py drives per-architecture converter classes in conversion/ registered via @ModelBase.register, writing GGUF through the gguf-py package (gguf-py/pyproject.toml, name=gguf v0.19.0); tensor name/layout tables live in gguf-py/gguf/constants.py.
  convert_hf_to_gguf.py#L15-25conversion/base.pygguf-py/pyproject.tomldocs/development/HOWTO-add-model.md
• The web UI is a SvelteKit + Vite + Tailwind app in tools/ui (package.json name=llama-ui) that is compiled to static assets and embedded into the server binary via a CMake-generated ui.cpp; LLAMA_USE_PREBUILT_UI downloads prebuilt assets from the HF bucket 'ggml-org/llama-ui' by default.
  tools/ui/package.jsontools/ui/CMakeLists.txt#L1-37tools/server/server-http.cpp#L5
• Top-level CMakeLists.txt is the single build system (cmake 3.14...3.28): version is LLAMA_VERSION 0.4.1-dev, feature options include LLAMA_BUILD_COMMON/TESTS/TOOLS/EXAMPLES/SERVER/APP/UI, LLAMA_OPENSSL, LLAMA_SUBPROCESS, LLAMA_LLGUIDANCE, plus deprecated-option shims mapping LLAMA_CUDA etc. to GGML_* options.
  CMakeLists.txt#L1-23CMakeLists.txt#L129-146CMakeLists.txt#L174-195
• Third-party code is vendored under vendor/ (cpp-httplib, nlohmann json, miniaudio, stb, sheredom subprocess.h, hash) with a dedicated CI workflow .github/workflows/check-vendor.yml verifying vendor integrity.
  vendor/CMakeLists.txt.github/workflows/check-vendor.yml
• Deployment shapes: static or shared libllama (BUILD_SHARED_LIBS default ON except MinGW/Emscripten), the `llama` unified binary, individual llama-* tool binaries, a WASM/Emscripten build (MEMORY64 enabled), Docker images via docs/docker.md, and an install-script self-update path gated on LLAMA_INSTALL_BUILD.
  CMakeLists.txt#L55-79app/llama.cpp#L27-41docs/build.md