CodeVetter · Repo Unpack · Data flow
ollama/ollama
Requests enter via Gin handlers -> model resolution (manifest -> Model struct) -> option merge (defaults < generation defaults < model < request) -> Scheduler -> runner subprocess over loopback HTTP -> streamed tokens parsed back into API responses. Model artifacts flow registry -> blob downloads -> manifest write; create flows Modelfile/safetensors dir -> layer blobs -> manifest. Chat text flows messages -> Go renderer or llama-server chat template -> prompt -> token stream -> Parser.Add splits content/thinking/tool calls -> NDJSON stream.
- Option precedence is layered in modelOptions: api.DefaultOptions -> model.GenerationDefaults (parsed from GGUF metadata) -> model.Options (Modelfile params) -> request options, with embedding models getting a NumBatch=2048 default capped by context
- Chat path in ChatHandler: ShouldBindJSON -> parseAndValidateModelRef -> getExistingName -> GetModel -> (unload shortcut for keep_alive=0) -> scheduleRunner -> renderer/parser selection -> llm runner Chat/Completion -> NDJSON stream to client
- GetModel resolves a name through manifest.ParseNamedManifest -> config blob decode (ConfigV2) -> layer iteration, grouping split GGUF shards via groupManifestSplitGGUFLayers; parseFromModel auto-pulls when the manifest is missing
- Prompt rendering branches: models with a Go renderer name go through renderers.RenderWithRenderer; others use template.Template (embedded .gotmpl set indexed by index.json) or llama-server's native chat_template via /v1/chat/completions
- Output parsing: streamed runner tokens are fed to Parser.Add which returns (content, thinking, toolCalls); PreservedTokens tells llama-server which grammar tokens must survive detokenization
- Pull flow: PullHandler -> PullModel -> registry layer resolution -> blobDownload.Run with chunked resumable parts -> sha256 verification -> manifest write; progress is fanned out as api.ProgressResponse over a channel to streamResponse
- Blob writes are content-addressed: CreateBlobHandler accepts POST /api/blobs/:digest and HeadBlobHandler probes existence; digest validity enforced by manifest.ValidateDigest regex sha256[:-][64 hex]
- Runner placement: scheduler loads GGUF metadata (llm.LoadModel), predicts VRAM via llm.PredictServerVRAM, selects GPUs via selectLlamaServerPlacement honoring OLLAMA_SCHED_SPREAD and OLLAMA_GPU_OVERHEAD, then evicts loaded runners if predicted footprint exceeds 80% of free memory
- llama-server subprocesses receive model path, ports, -ngl/-t only when explicitly set (otherwise auto-detect), media marker rewriting ([img-N]) for vision, and report memory back via parsed log lines into runner's vramByDevice/systemFreeAtLoad
- Desktop app persists chats/settings in sqlite (mattn/go-sqlite3) via app/store with a frozen v2 schema (schema.sql) plus migrations; the React UI talks to the same Ollama HTTP API
- Inference request logging is optional and bounded: withInferenceRequestLogging wraps /api/generate, /api/chat, and the /v1 inference routes, gated by OLLAMA_DEBUG_LOG_REQUESTS
- Context management: num_ctx defaults come from OLLAMA_CONTEXT_LENGTH or a VRAM-tier auto default (numCtxAuto), clamped to model train context; `shift` enables sliding-window context shift except for deepseek2 family; `truncate` drops history instead of erroring