Distilling a Claude Code session into a short brief looked like the perfect job for a local model. Private, free, offline. It lost to a script with no model judgement at all. 🤦♂️
Why I wanted this to work
This all started with graveyard, the tool I use to bury idle Claude Code sessions and resurrect them later. I wanted a “lossy” bury mode (some detail considered “noise” gets thrown away on purpose): take a dead session’s whole transcript, squeeze it down to the parts that matter (the decisions, what changed, the dead ends, what’s still open), and hand that to the resurrected agent instead of the whole raw thing.
I was willing for any model to do it, as I wouldn’t run it on every bury. But it seemed like a task that would fall squarely in the local model competency zone (a model running on my own Mac, not in someone’s cloud), so I was excited about the privacy aspect, as well as saving tokens/usage.
It started as a quick comparison
Honestly, the idea of a “gold set” was sort of sold to me (by Claude, while I was planning the graveyard’s lossy bury feature). Then I started thinking of more and more variables that I thought could dramatically change the outcome:
- the prompt the model gets
- pre-filtering the transcript first, and whether that helped or hurt compared to the LLM-compressed version
- System One models (e.g. Jev, or a small local one,
tev1) with a decision endpoint deciding “should this chunk be kept or not?” System One models are built to make one quick judgment call instead of writing text; a chunk is a slice of the transcript — those models tend to be extremely cheap and fast, so if they worked they’d be a viable alternative.
The variables just kept stacking up, to where a testing matrix (every combination of model × prompt × pipeline, run and scored the same way) made a lot of sense.
How the eval/test setup works
The fixtures (the test inputs). Four real Claude Code session transcripts, picked to be different shapes:
- a 23-turn cloud-console cleanup, the noisiest of the four (pasted logs, retries, caveats)
- a code-heavy skill build with 11 commits
- a 160-tool-call agentic build (a 6.2 MB transcript)
- a 36-turn machine cleanup
The gold sets (the answer keys). For each fixture, one model (Claude Opus 5.5) wrote an answer key:
- weighted must-survive facts (1–3, where 3 = “a resumed agent is lost without this”)
- noise that should not make it into the brief
- 10 questions a resuming agent would need answered
Then a second, independent Opus agent reviewed each gold set adversarially (told to hunt for mistakes, not to approve it). That review caught a stale commit hash, facts that contradicted the key’s own noise list, missing must-survive facts, and a fact describing the wrong system. Don’t skip this step, since every score rests on the answer key.
The judge (a model grading the other models’ work). The same Claude Sonnet prompt, every time, grades every brief against its gold set:
- recall: how much of what mattered made it in (the weighted share of must-survive facts the brief states)
- QA: the share of the 10 questions a fresh agent could answer from the brief alone. Sonnet plays that fresh agent: it answers each question using only the brief, then checks its answers against the key. So apart from Sonnet’s own briefs, no model grades a brief it wrote.
- hallucinations: the AI-world word for a model making stuff up (claims the brief makes that the key doesn’t back)
These roll into one overall score: overall = 0.5·recall + 0.5·QA − 0.1·min(hallucinations, 3), floored at 0. In plain words: half the credit for facts kept, half for questions answered, minus a tenth for each made-up claim (capped at three). Every score in this post runs 0 to 1, higher is better.
Grounding the hallucinations (checking them against the source). The judge only sees the answer key, so it flags any detail the key left out, true or not. So every flag gets re-checked against the actual transcript, and only real inventions count. Across the whole run, 206 of 473 flags (44%) turned out to be true details the gold set didn’t list… I guess the gold set wasn’t all that golden. Part of that is by design: it lists what must survive, not everything that’s true, so judging hallucinations against it was the wrong test. And part of it is the keys themselves: the review caught real errors in them, which makes me think it didn’t catch all of them. Treat these scores as relative (this setup beat that one), not absolute.
The baseline. A plain PHP “pre-filter” script, no AI model involved at all, so its output is deterministic (same input, same output, every time). It throws out harness noise (the Claude Code bookkeeping stuff), keeps one copy of anything pasted twice, trims long tool output, and lists the hard facts it can pull straight from the tool calls (files edited, commits, commands that worked). If a model can’t beat that, why pay for it?
What got varied:
| Variable | Options |
|---|---|
| Model | none (the baseline script) · qwen2.5-coder:1.5b · gemma4:e4b · llama3.1:8b · qwen3.5:9b · gemma4:26b-nvfp4 · Claude Haiku · Codex gpt-6-luna (at its lowest reasoning setting) · Claude Sonnet (the priciest; I expected it to be the best-case reference, not a contender) |
| Prompt | terse bullets · fixed headings · “brief an engineer taking over tomorrow” |
| Pipeline (how the transcript gets fed in) | pre-filtered or raw · one call · chunk-and-merge · rolling refine · truncated |
| Chunk gate | none · TypeSafe Jev · SubQ subq-snap-preview · a small local stand-in |
The local model names are family:size. The number is the parameter count in billions (roughly: bigger is smarter and hungrier for memory). e4b is Gemma’s small “effective 4B” model, and nvfp4 is a compressed format that lets the 26B fit in memory.
The pipelines, in plain words:
- pre-filtered or raw: run through the baseline script first, or fed in as-is
- one call: the whole transcript in a single request
- chunk-and-merge: split into ~12,000-character chunks, summarize each, then merge the summaries
- rolling refine: keep one running brief and update it chunk by chunk
- truncated: cut down to 12,000 characters to fit a small model’s context window (how much text it can read at once)
- chunk gate: a cheap yes/no model that decides whether a chunk is worth summarizing at all
What I found
Local models didn’t beat the script
How to read the numbers: every score runs from 0 to 1, higher is better. Think of it as a percentage.
- Recall is how much of what mattered made it into the brief. The script’s 0.22 means its brief kept about a fifth of the must-survive facts.
- QA is how many of the 10 questions a fresh agent could answer from the brief alone. Sonnet’s 0.73 means about seven of them.
- Made-up claims is how many things, on average, a brief stated that never happened in the session.
- Overall is the one-number score from earlier: recall and QA averaged, minus 0.1 per made-up claim (up to three). That penalty is why a model can keep more facts than the script and still score lower overall.
| Model | Overall | Recall | QA | Made-up claims | Cost/brief | Time/brief |
|---|---|---|---|---|---|---|
| no model (baseline script) | 0.22 | 0.22 | 0.23 | 0 | $0 | 0s |
| Sonnet, one call, whole transcript | 0.63 | 0.68 | 0.73 | 0.8 | ~$0.29 | 30s |
| Haiku, one call, whole transcript | 0.36 | 0.54 | 0.55 | 2.3 | ~$0.04 | 24s |
| Haiku, chunk-and-merge | 0.29 | 0.56 | 0.59 | 5.3 | ~$0.05 | ~6 min |
| gemma4:26b | 0.15 | 0.38 | 0.38 | 2.3 | $0 | 287s |
| qwen3.5:9b | 0.13 | 0.32 | 0.35 | 3.5 | $0 | 261s |
| llama3.1:8b | 0.10 | 0.35 | 0.30 | 3.9 | $0 | 157s |
| gemma4:e4b | 0.07 | 0.20 | 0.21 | 1.6 | $0 | 183s |
| qwen2.5-coder:1.5b | 0.03 | 0.14 | 0.11 | 3.1 | $0 | 41s |
The local model rows average every pipeline each one ran; most of their runs were chunk-and-merge.
The briefs the bigger local models wrote held more of the must-survive facts than the script’s did, according to the Sonnet judge (recall 0.32–0.38 vs 0.22). But those same briefs also stated two to four things that never happened in the session, and that penalty dragged their overall score below the baseline. The inventions were the kind a resumed agent would act on, e.g.
- miscounting how many services got disabled
- turning a decision the session made into an open to-do to undo it
- marking finished work as still open.
Their best single runs did edge past the script on the cleanup session, though single runs turned out to be shaky ground.
A great first run can be a lucky draw
gemma4:e4b scored 0.42 on its first run, at temperature 0 (the setting that makes a model pick its most likely answer every time, so the run is repeatable). That’s its best number in the whole eval, better than the script. Re-run with a little randomness turned back on, it scored 0.24 and 0.22. If something looks good, run it again before you believe it.
The judge itself was stable: the same brief graded three times moved at most 0.05, and usually not at all. So the swings came from the model.
One call on the whole transcript beats chunk-and-merge
Same model (Haiku), same four sessions, two ways of feeding it the transcript. These are the two Haiku rows from the table above:
| Pipeline | Overall | Made-up claims | Time/brief |
|---|---|---|---|
| one call, whole transcript | 0.36 | 2.3 | 24s |
| chunk, summarize, merge | 0.29 | 5.3 | ~6 min |
Cost was about the same, a few cents either way. Chunking kept about as many facts, but it more than doubled the made-up claims: every chunk summary is another chance to invent something, and the merge step carries those inventions forward.
Truncating to fit kills everything
On the cleanup session, Sonnet scored 0.80 with the whole transcript and 0.16 when it was cut to 12K characters. If your model can’t fit the whole transcript in its context window, cutting it down to fit just hides the part with the answer.
The chunk gates didn’t help
The idea is that before the (slower, pricier) summarizing model reads a chunk, a cheap, fast model takes a quick look and answers one question: “is anything in here worth remembering?” If not, that chunk gets skipped, and you save the time/money of summarizing it.
Each gate answers with a confidence score from 0 to 1, and you pick the cutoff: any chunk scoring below it gets skipped. I tested this on the cleanup session, split into 7 chunks and run four different ways, so 28 keep-or-skip calls per gate. At the cutoff I shipped with (0.3), all three gates said “keep” to all 28 chunks. Nothing got skipped, so nothing got saved.
So I replayed the two real gates’ scores at stricter cutoffs to see what they’d have done (the third was a small local stand-in for Jev, there only as a placeholder):
- Jev rated every chunk about the same (somewhere between 0.83 and 0.90), boring or important, so it couldn’t tell them apart. At a 0.9 cutoff it skipped 27 of the 28.
- SubQ was the only gate that rated some chunks lower than others. But the chunks it rated lowest included the one holding the session’s final outcome, which is the thing a resumed agent most needs to know. At a 0.8 cutoff, that fact disappeared from the brief.
That’s one session, with no effort spent tuning the cutoffs, and Jev and SubQ are both preview products. So all this shows is that they didn’t help here.
Even the best model keeps about two-thirds
Sonnet, on the whole transcript, kept about 68% of the must-survive facts. That’s good enough for a first read, but not good enough to throw the transcript away.
Two more judges agreed on the order
Sonnet grading its own briefs is an obvious problem, so I re-graded the 12 whole-transcript briefs (Sonnet’s and Haiku’s) with two more judges: Claude Opus, and Codex (gpt-6.1-sol, a non-Claude model, so no family loyalty).
| Judge | Sonnet overall | Haiku overall | Sonnet’s lead |
|---|---|---|---|
| Sonnet | 0.63 | 0.36 | +0.27 |
| Opus | 0.64 | 0.50 | +0.14 |
| Codex | 0.38 | 0.23 | +0.15 |
Sonnet came out on top under all three. But its lead was about half as big under the other two judges, so some self-preference was probably baked into the original numbers.
What surprised me more was that the judges mostly agreed on recall (how much made it in) and mostly disagreed on made-up claims. Grading the same Haiku briefs and checking against the transcript, Opus found about 0.6 invented claims per brief, Sonnet found about 2, and Codex found almost 4. So the absolute numbers in this post depend a lot on who’s grading; the order doesn’t.
My picks, in order
These are my tradeoffs for graveyard, where a brief gets written once per bury and has to be trusted later. Yours may differ.
- Sonnet, one call on the whole transcript. It kept the most and made up the least of any model (under one invented claim per brief). At ~$0.29 a brief it’s the priciest, but I wouldn’t run it on every bury anyway. The asterisk is that Sonnet was also the main judge, though Opus and Codex ranked it first too.
- Haiku, one call on the whole transcript. About 24 seconds and 4 cents a brief, but it made up more than Sonnet under every judge (anywhere from under one to about four claims per brief, depending on who’s grading). If I were condensing every bury, this is where I’d land.
- The no-model script. Free, instant, and it never makes anything up, but it only keeps about a fifth of what matters. It’s the fallback that always runs, and the floor for anything else.
No local model made the list for this job.
So what are local models for?
The answer here is “don’t use the local model,” and that really bummed me out. Often when running these kinds of comparisons, that’s the answer, and it bums me out as it’s hard to see what the point of a local model is.
I wouldn’t say they are useless, but they definitely have very limited uses. I use them for things like:
- a quick summary of a cloud conversation or session that’s in the works, so that I can save tokens, and it does a reasonable job
- commit messages, where it wraps up the current context of the commit and creates a message (some of the local models do a decent job of that)
to-text, a bin command I recently created that converts images to text (I’ve only tested it with very small screenshots with minimal text, so not sure how well it would do with larger text)
So it’s not that they have no use. I’m just often surprised at the limits they have.
Then why does that first one work?
A local model summarizing a session is exactly what this eval tested. But asking an agent to review the last few threads to summarize where we’re sitting is much different than asking it to review an entire transcript and all the tool calls, and then distill it down to only the most important bits. That second one is the exact task I’m trying to build. And again, local models seem to be pretty poor at that job.
They do invent things in the quick summaries too, occasionally. But it’s much easier to catch it there, because I was just in that session. With the burying thing, by the time I come back to resurrect a session I’m going to have forgotten it. It’s going to be very hard for me to verify at that point if the information it kept was hallucinated.
What I’m building instead
It did change what I want to build. Graveyard will store a compressed version of the full transcript and the smaller distilled version. When I resurrect, it resurrects based on the distilled version. And if I have deeper questions that it can’t answer, it reaches into the uncompressed version to find the answer.
Run your own
I think it’s worth running these tests, especially with different variables: different prompts, different ways to package the information for the LLM to grok. Don’t be surprised if the results are underwhelming, but one surprising positive result is worth it most of the time.
I turned this into a small blueprint repo: model-eval-blueprint. It’s one script, the prompts, and a guide your agent can follow to run model evals on your own job with your own models. Point your agent at it and tell it what job to test and which models to compare.
If you build your own, here’s what I’d keep:
- A no-model baseline. It’s the bar every model has to clear.
- Gold sets reviewed by a second, independent model. It caught real errors in the keys.
- Grounded hallucination checks. Re-check every flag against the source, or you’ll overcount by a lot.
- Re-runs (replicates) of anything that looks good. A single good run can be luck.
- A clean prompt for every CLI backend.
codex execinjects your global~/.codex/AGENTS.mdand your skills list into every call (about 35KB here), even with--ignore-user-config. The fix was a throwawayHOMEplus aCODEX_HOMEholding only a symlinkedauth.json, which got it down to about 13.5K tokens of stock Codex overhead. Check yours withcodex debug prompt-input.claude -p --setting-sources '' --system-prompt …is close, but it still adds an environment block (working directory, git status, date), so run it from a neutral directory like/tmp. In one test run, a Haiku brief listed “Is a git repository: false” as an open item.
But do keep in mind these models can get your fan spinning, ESPECIALLY when running a matrix test. My machine was pinned for hours during these tests! The biggest one, gemma4:26b-nvfp4, sat at 18.6–24.5 GB of RAM and took 2–10 minutes per brief. (It also “thinks” silently before answering by default, and that thinking used up its entire reply limit, so it returned an empty brief until I ran it with think: false.) The full local sweep across the other sessions would have taken about 17 more hours, so I cut it once the first session showed local ≈ baseline.
The caveats
- Four fixtures, all mine.
- Most cloud setups ran once.
- Codex and the gates ran on one fixture only.
- Sonnet was the main judge. Opus and Codex re-graded only the 12 whole-transcript Sonnet/Haiku briefs; the local models and the script were graded by Sonnet alone.
- Nothing bigger than a 26B local model was tested, and that one ran just 3 times.
I would definitely be willing to try it again, and I’m not going to give up on trying local models for different tasks. Hopefully the next generation keeps improving the jobs they’re able to do on the hardware I already have. Until then, the script stays in the loop, and a cloud model does the condensing.