CodeVetter · Repo Unpack · Extension points
ggml-org/llama.cpp
Primary extension seams: new model architectures (conversion/ ModelBase.register + src/models/*.cpp + llama_model_mapping switch + gguf constants), new ggml backends (ggml_backend_reg_t + dlopen registry), new samplers (llama_sampler interface + common_sampler_init chain), new chat/tool parsers (common/parsers + PEG/jinja), new CLI args (common_arg table), new server endpoints (ctx_http.get/post + server_routes handlers), new built-in tools (server_tool subclasses), and new unified-binary subcommands (cmds[] table).
- New model architecture recipe is documented: register a TextModel/MmprojModel subclass via @ModelBase.register in conversion/, add MODEL_ARCH + MODEL_TENSORS entries in gguf-py/gguf/constants.py, add LLM_ARCH_* enum, a src/models/<arch>.cpp graph builder, and a case in llama_model_mapping — see docs/development/HOWTO-add-model.md.
- ggml backends are plugins: a backend exports a ggml_backend_*_reg() registered statically under #ifdef GGML_USE_<X> or loaded at runtime by ggml_backend_load(path) which dlopens a shared library — GGML_BACKEND_DL builds each backend as a separate .so.
- Samplers implement the llama_sampler interface (src/llama-sampler.cpp provides ~20 stock samplers); common_sampler_init assembles them into a chain in a data-driven order, so adding a sampler means a new llama_sampler_init_* plus a slot in the ordering switch.
- Chat/tool-call parsing is extensible at three layers: the standalone jinja engine (common/jinja) for templates, per-model PEG parsers in common/parsers/*.cpp (one file per model family), and chat-auto-parser which auto-detects model-specific features — all surfaced via common_chat_templates_apply/common_chat_parse.
- CLI arguments are declarative: each option is a common_arg entry in common_params_parser_init with flags, env var (set_env), applicable examples (set_examples/set_excludes), and handler lambda; get_common_arg_defs exposes the table for introspection.
- Server endpoints plug in via ctx_http.get/post/del + a handler on server_routes (routes registered in llama_server); all handlers are wrapped by ex_wrapper which maps std::invalid_argument to 400 and other exceptions to 500 with an OAI-style error body.
- Built-in server tools extend the server_tool interface (name/schema/run) and are appended in the tools registry (tools.push_back(std::make_unique<server_tool_...>())); an isolate-backed tools_io abstracts file/exec ops for sandboxed runtimes.
- New unified-binary subcommands: add an entry to the cmds[] table in app/llama.cpp (name, desc, aliases, hidden flag, func) — hidden commands exist but aren't listed in default help.
- KV/memory is abstracted behind llama_memory_i (seq_rm/seq_cp/seq_keep/get_can_shift/apply); multiple implementations already coexist (llama-kv-cache, -iswa, -dsa, -dsv4, -msa, memory-hybrid, memory-recurrent) selected per architecture.
- Structured output plugs through grammars: standalone .gbnf files in grammars/, JSON-Schema->GBNF conversion, and an optional llguidance backend (LLAMA_LLGUIDANCE) via llama_sampler_init_llg in the sampler chain.
- Speculative decoding types are enumerable and extensible: common_speculative_type_from_name / _from_gguf map names and GGUF metadata to enum common_speculative_type (draft, MTP, ...), implemented in common/speculative.cpp (~3k lines).
- MCP servers extend tool surface without code changes: server_mcp_server_config::parse_from_json accepts Cursor-style 'mcpServers' JSON to spawn stdio MCP servers whose tools are listed via list_tools and invoked via call_tool.