← Back to overview

Imprint benchmark evidence

Measured 2026-10-23 with Claude Code CLI: 180 trials, 90 complete successful pairs and 0 failed trials. Both modes receive identical public source files and synthetic historical evidence. ON adds read-only Imprint tools and memory-first guidance with file fallback. Overview totals sum paired trials; task tables report medians across 5 repetitions. Sanitized evidence is at benchmark/published/20261023T140134777248Z-fdea98ae-s1729/.

Tokens
+25.0%
2,544,880 → 3,182,015
Cost
-9.0%
$4.1253 → $3.7542
Environment
OS: Linux
Python: 3.12.15
Model: claude-sonnet-5-5
Imprint: ee040e61a0be
Memories: 297

Reading the numbers

Actual sums across complete successful pairs; cache input included. Per-task rows are medians, not summed component medians. Claude Code reported list-price estimate, not a subscription invoice Provider-managed cache; measured separately, not forced cold. Concurrent trials may share cache. No answer-quality uplift is claimed. Lexical reference checks do not establish factual correctness, and follow-up turns were not measured. Protocol v2 uses a different corpus and model from the old Imprint 0.5.1 experiment, so this is a fresh baseline rather than a controlled version comparison.

Uncertainty and latency

Token change, 95% interval: 11.3% to 43.6%. Cost change: -19.0% to 2.9%. End-to-end p50: 12.5s OFF → 13.9s ON; p95: 17.3s → 17.5s. Intervals resample paired tasks and repetitions, not individual tokens, and describe this fixed suite. The cost interval includes no change, so the point estimate does not establish a clear saving.

Before and after compact MCP responses

RunTrialsToken change ON vs OFFEstimated cost change ON vs OFF
Previous response policy180 +31.7% (18.5% to 45.7%) -5.6% (-20.6% to 10.5%)
Compact response policy180 +25.0% (11.3% to 43.6%) -9.0% (-19.0% to 2.9%)

Same frozen evidence, task suite, model and repeat count. These runs were collected sequentially, with provider-managed caches and service variation. Each row compares matched ON/OFF trials within its run; this table is descriptive and does not isolate a causal version effect or establish answer quality. Previous trial evidence.

Retrieval on the labeled public corpus

ModeRecall@5MRR@10nDCG@10Warm p50Warm p95
semantic0.802 0.8690.782 107.5ms120.8ms
hybrid0.833 0.8390.776 151.4ms173.0ms

Distinct gold sources in the first k returned chunks; duplicate source gains are zero. 42 positive queries, plus explicit filter/workspace boundary cases; 5 timing repetitions per mode. Model load and first query: 1.4s. Fixed passages isolate retrieval from ingestion quality. These metrics score ranked chunks before MCP response budgeting; they do not prove which evidence an agent read or generated-answer correctness.

Hybrid − semantic Recall@5: 3.2 percentage points; paired query-family bootstrap interval: -1.2 to 8.3 percentage points. The interval includes no difference; this does not establish a clear retrieval winner. Boundary checks: 60/60 passed.

onnx-community/Qwen3-Embedding-0.6B-ONNX, 1024 dimensions, 2048-token cap, 2 ONNX threads. Host: 13th Gen Intel(R) Core(TM) i9-13900HX. Timings come from a shared Linux host and are not platform-wide performance guarantees.

Runtime: ee040e61a0be; frozen evidence: fdea98aea4bb. Methodology, corpus and limitations · Trial evidence and manifest

Information

tokens +20.0% cost -15.8%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Chunking system 25,978 38,714 +49.0% $0.0539 $0.0527 $-0.0013
Embedding model 26,552 38,225 +44.0% $0.0559 $0.0443 $-0.0116
Workspace isolation 58,649 52,836 -9.9% $0.0656 $0.0593 $-0.0063
Metadata tagging 37,501 32,961 -12.1% $0.0508 $0.0387 $-0.0121
Qdrant lifecycle 28,444 35,080 +23.3% $0.0647 $0.0490 $-0.0157
Portable device context 22,928 34,630 +51.0% $0.0368 $0.0394 $+0.0025
Scoped source purge 25,368 37,975 +49.7% $0.0477 $0.0475 $-0.0002

Decision Recall

tokens +9.4% cost -4.3%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Qdrant vs embedded 21,031 20,499 -2.5% $0.0285 $0.0267 $-0.0018
LLM tagger default off 21,170 21,021 -0.7% $0.0337 $0.0287 $-0.0050

Debugging

tokens +18.2% cost -10.0%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Search returns nothing 24,425 32,931 +34.8% $0.0437 $0.0381 $-0.0055
Slow embedding on first run 45,176 39,643 -12.2% $0.0687 $0.0638 $-0.0049
Encrypted sync boundary 32,298 41,722 +29.2% $0.0746 $0.0697 $-0.0049

Cross-Project

tokens +22.9% cost -24.2%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Async work pattern 24,367 34,528 +41.7% $0.0463 $0.0447 $-0.0016
Config precedence 26,425 31,229 +18.2% $0.0382 $0.0338 $-0.0044

Session Summary

tokens +63.5% cost +42.0%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Recent activity 10,742 17,590 +63.7% $0.0063 $0.0081 $+0.0018

Creation

tokens +55.5% cost +28.0%
Prompt OFF tokens ON tokens Δ % OFF cost ON cost Δ $
Qdrant connectivity test 20,223 55,307 +173.5% $0.0313 $0.0573 $+0.0260
Memory count utility 19,521 22,811 +16.9% $0.0234 $0.0349 $+0.0115
Workspace stats CLI 30,675 31,195 +1.7% $0.0360 $0.0297 $-0.0063

How to reproduce

# Full suite: 18 prompts × 5 repeats × 2 modes
bash benchmark/run.sh

# Subset (one prompt per category)
bash benchmark/run.sh --subset

# Filter by category or prompt
bash benchmark/run.sh --category debug
bash benchmark/run.sh --prompt info-1

# Summarize existing results
python3 benchmark/summarize.py benchmark/results/RUN_ID --publish

What we're measuring

  • Total tokens: input + output + cache_read + cache_create across every reported model. Subagents are disabled in this controlled suite.
  • Cost: Claude Code's reported total list-price estimate. Cache classes are retained separately; this is not a subscription invoice.
  • Latency: end-to-end client wall time, including MCP startup for ON; serial warm retrieval timings are reported separately.
  • Quality: labeled source retrieval metrics and diagnostic reference checks. A biased LLM judge is no longer used.