Local helper models

The default optional helper is Qwen3.5-4B, Apache-2.0, using Unsloth’s Q4_K_M GGUF. Text-only inference needs no vision projector. Its 2.74 GB file is pinned to revision e87f176479d0855a907a41277aca2f8ee7a09523 with a source-controlled SHA256. First use needs internet; subsequent inference uses local files. Custom GGUF repositories retain their configured downloads and should be pinned/verified by their operator.

Embeddings, ordinary search and explicit MCP stores do not require this helper. LLM tagging and session summaries remain opt-in, and their prompts are redacted before any provider sees them (secret redaction). The dashboard loads the model only when chat is invoked. The helper runs on llama.cpp’s binding, llama-cpp-python, which every installation includes because embeddings run on it too (ADR 0086). The first local chat only downloads the model if it is missing, and shows the progress in the chat (ADR 0035). A binding that does not load is a damaged installation; chat says to run imprint bootstrap --non-interactive. Windows x64 and Linux x64/arm64 install upstream’s prebuilt wheel and macOS the prebuilt Metal wheel this repository publishes (ADR 0090); other platforms build it from source, which needs a C/C++ toolchain.

Enable the roles you want

imprint config set chat.provider local
imprint config set tagger.llm_provider local
imprint config set tagger.llm true
imprint config set summarizer.provider local
imprint config set summarizer.enabled true
imprint config set recall.llm true          # uses tagger.llm_provider when it is local

To replace previously persisted helper choices, set both chat.model_repo and tagger.local.model_repo to unsloth/Qwen3.5-4B-GGUF, both chat.model_file and tagger.local.model_file to Qwen3.5-4B-Q4_K_M.gguf, and both context settings (chat.n_ctx, tagger.local.n_ctx) to 8192. Reset any explicit chat.model_path / tagger.local.model_path override if you want managed downloads. Existing user choices are preserved until explicitly changed. Restart processes after editing settings.

All four roles use one cached GGUF file by default. Chat and tagging reuse the same model object within a process when path, context and GPU-layer settings match. Summaries and recall use tagger.local.* settings; recall loads the model in the MCP server process, never downloads it, and uses only a model on this computer (configuration). A shared inference lock serializes local streaming and classification. Different configurations or separate dashboard/CLI/MCP processes can still load separate instances. For sharing across processes, select an existing Ollama/vLLM endpoint explicitly; no new server is installed or started automatically.

A larger chat model

chat.local_model picks the local chat model in Settings → Chat (ADR 0153):

• Qwen3.5 4B (default): the shared helper above, through chat.model_repo, chat.model_file and chat.model_path.
• Qwen3.5 35B-A3B (MoE): unsloth/Qwen3.5-35B-A3B-GGUF, Apache-2.0, Q4_K_M, 22.0 GB, pinned to revision bc014a17be43adabd7066b7a86075ff935c6a4e2 with a source-controlled SHA256. About 3B of its 35B parameters run per token, so it is about as fast per token as the 4B but answers better. It is offered on computers with about 32 GB of memory or more; elsewhere the choice is shown as unavailable with the reason. A GPU that cannot hold it runs it on the CPU (ADR 0087); lower chat.n_gpu_layers to keep part of it on the GPU.
• Custom: a GGUF path on this computer, <owner>/<repo>/<file>.gguf, or a Hugging Face file link. Repository files download into data/models/<owner>--<repo>/ from main, unverified.

Only chat changes: tagging, summaries and recall keep tagger.local.*. The next chat loads the new model and lets the previous one go unless the tagger uses the same file.

imprint config set chat.local_model qwen3.5-35b-a3b
imprint config set chat.local_model bartowski/Qwen3-14B-GGUF/Qwen3-14B-Q4_K_M.gguf
imprint config reset chat.local_model

Bounded generation

The default context is 8192 tokens. Helper GPU layers default to -1 (all supported layers). On the gpu and auto profiles setup installs llama.cpp’s Vulkan build on Windows and Linux x64 (Metal on macOS) when it finds a GPU, so the helper offloads without a CUDA toolkit (ADR 0049). Each model runs whole on one GPU, the one model.gpu_device selects (auto: the discrete GPU with the most memory), rather than split across several; a GPU that cannot load it leaves it on the CPU with a note on standard error (ADR 0087, ADR 0086). Embeddings use the same GPU when model.device allows, so the helper and the embedding model share its memory. Set both chat.n_gpu_layers and tagger.local.n_gpu_layers to 0 for CPU helper inference. Context, runtime buffers and other applications add memory beyond quantized file size. Larger contexts and a 9B helper remain measured opt-in choices.

Local helper inference uses four CPU threads and a 256-token prompt batch. An opt-in native smoke is available as PYTHONPATH=. <venv-python> scripts/smoke-local-helper.py --model-path /path/to/Qwen3.5-4B-Q4_K_M.gguf. It verifies the pinned checksum, sharing, non-thinking template, structured classification/summary and a dashboard tool round trip with mocked storage, in a temporary data directory. Without --model-path, it downloads the model into that temporary directory. This is a small functional check, not a quality benchmark.

The embedded GGUF chat template is rendered with enable_thinking=false. This replaces the ineffective /no_think text prefix for local Qwen3.5. Missing or incompatible templates fail at load. Summaries and recall plans use a JSON-schema grammar; for recall plans it was no slower than unconstrained output on CPU, which spends more tokens on pretty-printed JSON (ADR 0058). Classification sends its prompt unconstrained, because on a GPU the grammar cost more than the inference, and its reply is validated like a remote provider’s (ADR 0049). Both are followed by existing application validation. Valid JSON does not establish factual correctness.

The classification prompt asks for the JSON on one line, which takes under half the tokens of indented JSON (ADR 0163). Unlike chat’s, its instructions are not kept evaluated between memories: they follow each memory’s project and file, because with them first the model classified differently (ADR 0164). With tagger.llm_concurrency above 1, a second context on the same model decodes that many memories as one batch, each in its own 4,096-token sequence (ADR 0165).

Chat calls tools through each model’s native protocol (ADR 0059). One registry holds every tool’s JSON schema. Locally, the schemas, earlier calls and tool results are rendered by the GGUF’s own chat template. The reply is parsed in the call format of the model’s family, chosen by chat.tool_format (Settings → Chat → Tool-call format; ADR 0061):

Value	Models	A call in the reply
auto (default)	Detected from the loaded GGUF’s template
qwen-xml	Qwen3.5, Qwen3-Coder	<tool_call><function=NAME><parameter=P>value</parameter></function></tool_call>
hermes	Qwen2.5, Qwen3, Hermes	<tool_call>{"name": …, "arguments": {…}}</tool_call>
llama3	Llama 3.1–3.3	the whole reply is {"name": …, "parameters": {…}}, optionally after `<
mistral	Mistral, Mistral Nemo	[TOOL_CALLS][{"name": …, "arguments": {…}}]
none	Any	the chat runs without tools

auto asks the template itself: it renders a probe tool call the way its model was trained to write calls, and the format whose parser reads that back is chosen. A template that renders no call any format reads (no tools support, or an unknown grammar) chats without tools, and the conversation’s first reply shows a notice naming the setting; choose a format there if the model can call tools. Each parser accepts only its family’s documented shape. Arguments are then typed by each tool’s schema, for every format and for remote providers: a string given for a number or boolean parameter is read as JSON, and string values stay verbatim, multi-line included. A value of the wrong type is returned to the model as a tool error. Generation decodes special tokens as text, because markers such as Mistral’s [TOOL_CALLS] are special tokens. A template that takes tools other than through the standard tools variable, such as Phi-4-mini’s, is not supported, and neither is a model whose tool template is stored separately (tokenizer.chat_template.tool_use, as in Hermes 3). Remote providers use their API’s function calling: tools, streamed tool_calls and tool messages. A vLLM server must run with --enable-auto-tool-choice and a --tool-call-parser for its model. The Anthropic provider uses its OpenAI-compatible endpoint https://api.anthropic.com/v1/.

Chat checks actual formatted prompt tokens, tool schemas included (about 1,700 tokens with Qwen3.5-4B’s template), and fits the prompt by dropping earlier history a piece at a time: earlier turns’ tool exchanges first, oldest first, then whole earlier exchanges, oldest first, so the conversation itself stays as long as it fits (ADR 0140). Once the conversation’s own text no longer fits, its oldest exchanges are summarized instead of dropped (see long conversations). Oversized current questions and evidence use the sequential section analysis below. Summaries keep a 5000-token estimated transcript budget, then cap actual transcript tokens to the local context while preserving instructions and recent transcript content. Ordinary classification has a bounded input and rejects context overflow. Research uses reply-wide time/token budgets and evidence-novelty tracking, with no fixed tool-round cutoff. Defaults are 300 seconds, 65,536 prompt-plus-output reservations and four consecutive read rounds without new evidence. Section workers share these budgets. Local prompt counts use actual tokenization; remote counts are estimated, and maximum output is reserved conservatively. The tools-free final answer has separate output space and a 120-second deadline. Deadline checks are cooperative; native prefill and an executing blocking tool may overrun. If final synthesis fails, Chat returns labelled evidence excerpts, while voice uses a brief spoken fallback. See ADR 0102. Voice mode offers only the read tools. Dashboard search forwards source/mode/offset filters and uses the workspace-specific wake state.

How chat researches

Chat and Echo answer from memory with a tool loop: the model calls tools, reads their results and calls more until it can answer.

• Read tools. search, get_memory (one memory in full, by the ID a result shows), neighbors, graph_scope, kg_query, list_sources, file_summary, file_chunks, read_source, status and wake_up. Every read tool that a result suggests (for example “expand with get_memory(…)”) is one chat can call; a test keeps it that way.
• Filters are hints. The model often adds a domain, layer, lang, kind or type filter it guessed. A memory not tagged that way would drop out however well it matches, so chat’s search also lists up to three results the filters left out, under “Also matching without …” (ADR 0160).
• No repeated reads. A read call repeated with the same arguments in one reply is not run again; the model reads a note pointing to the earlier result, and it counts as no new evidence (ADR 0160).
• Asking and scheduling. ask_user pauses the reply on a question (ADR 0159); the reminder, timer and scheduled-job tools work in Chat and Echo (ADR 0158).
• Reused instructions. The current time follows the question rather than sitting in the instructions, so the instructions and conversation stay an unchanged prompt prefix between turns. Qwen3.5 is a hybrid model that can only reuse a prompt that extends its last one exactly, so the local model keeps the evaluated instructions and tool definitions in memory (up to two checkpoints, about 170 MB each for Qwen3.5-4B) and restores them at the next turn; on a 4-thread CPU a new turn’s first request started in 2.6 s instead of about 90 s (ADR 0161).
• Budgets. Research ends when it stops finding new evidence (chat.research_no_progress_rounds), at chat.research_timeout_sec, or when chat.research_token_budget is spent; each request is charged for its new prompt and the output it generates (ADR 0157). The answer is then written from what research found.

Long conversations

Chat and Echo’s chat route keep the whole conversation available to the model in two ways (ADR 0141):

• A running summary. At the start of a reply, if the conversation’s text (earlier research aside) no longer fits beside the current summary, the oldest exchanges are summarized: the model writes one bullet per newly covered exchange, with message numbers, and the bullets are appended to the summary so far (at most 2,000 characters; the oldest bullets go first). It covers enough that the newest exchanges fill at most half the room left, so the next summary waits several turns. The summary sits in the system prompt under “Earlier in this conversation”, with the covered message range and a list of covered material such as long pasted messages and tool results. It is written at most once per reply, before research, with 300 output tokens and 120 seconds (the wait for the local model included), outside the research budget; the reply shows “Summarizing the earlier conversation…” meanwhile. If the model fails or runs out of time, the reply drops old exchanges as before and the next attempt waits six messages. Secrets are redacted in what the summarizer reads and writes. Chat stores the summary with the session; Echo stores it with the conversation’s chat while saving is on, otherwise for the voice session only (ADR 0146). On the maintainer’s Apple M3 with Qwen3.5-4B a summary took 19 to 62 seconds.
• conversation_history. A read-only tool, offered whenever the conversation has earlier messages, searches every earlier message of this conversation (the user’s, the assistant’s and tool results, summarized or not) by words, or reads one by number, 4,000 characters at a time from a character offset. A long pasted message shows up to three matching places. It sees only the current conversation. The tool strategy tells the model to use it, rather than memory search, when the user refers to something earlier in the conversation that is not shown.

Large chat questions and evidence

chat.analysis_workers defaults to 2. Change it under Settings → Chat → Sequential analysis workers, with the CLI (imprint config set chat.analysis_workers 3), or through IMPRINT_CHAT_ANALYSIS_WORKERS. Values 1–8 select logical workers, always run one at a time. 0 / Off disables section analysis and restores the 4,000-character tool-result clipping policy; an oversized current question then fails with a context error.

Short questions follow the ordinary chat path. Once current input or tool evidence exceeds the coordinator’s budget, workers take turns reviewing sections and updating their own bounded notes. Each worker uses a fresh prompt with a short question excerpt, its previous notes and the next section. Section sizes include all prompt overhead and output space, with a maximum 4,096-token worker context and 64 characters of overlap. Splits prefer newline or word boundaries. All sections must be processed before synthesis; the worker count is not a cap on the number of sections. The coordinator gets the reports and synthesizes the answer through its existing tool loop. ADR 0093 records the decision.

Workers use the same selected model/provider as Chat, without tools or additional model downloads. Locally, prompts are measured with the loaded model’s actual tokenizer/template. Remote chat uses chat.n_ctx as an analysis budget and a UTF-8-byte-based token estimate; it cannot verify the remote tokenizer or actual server context limit. Set chat.n_ctx for that provider if the budget needs to change. Both routes reserve the configured chat.max_tokens for synthesis.

When section analysis has captured an original, a tool-capable coordinator also gets read_chat_section. It reads bounded exact character ranges using the reply’s unique source IDs. Those originals exist only during that reply and are never exposed as MCP/global tools or file access. With section analysis enabled, new tool results are no longer blindly clipped at 4,000 characters: they are passed whole if they fit, or analyzed across their sections. Existing chat transcript storage still records original user messages and tool results; worker notes and progress are transient.

Chat and the same-chat voice route show section progress. Workers have a 120-second generation budget, at most 384 output tokens per set of rolling notes, and a shared 64-section limit per reply. A source is capped at 1,000,000 characters. A worker failure, exhausted section budget or findings that cannot fit the coordinator produces an explicit error instead of silently dropping unread sections. Cancellation closes the current token stream before another section starts; native prefill and model calls already holding the inference lock retain their existing interruption limits.

This is lossy analysis and adds inference time; it does not enlarge the final model’s context or establish answer accuracy. The coordinator is directed to preserve uncertainty and reread originals for exact details. Models without a supported tool format can use analysis but cannot call the reread tool. The small LFM voice route does not use this chat pipeline.

Remote provider configuration remains supported. Non-thinking template control here applies to the in-process runtime. LLM tagging also asks OpenAI-compatible servers not to think (chat_template_kwargs for vLLM and an openai provider at another address, reasoning_effort: "none" for Ollama); other uses of remote servers configure thinking at the server (ADR 0166). An external model is not automatically equivalent to the pinned GGUF.

See model selection rationale, tagging and ADR 0010 for validation and remaining evaluation work.