drona23/claude-token-efficient

Read both. The p_step^N framework lines up with what our N=5 semantic eval showed.

In SEMANTIC.md from #10, judge accuracy and completeness are 100% in both conditions across haiku, sonnet, opus. Em-dash and verbosity markers move; correctness does not. The rules change tone, not capability. That matches your point: p_step is set by what's dense in training, not by the system prompt.

The verification-on-same-weights observation is the cleanest framing I've seen for why "check your work" rules in CLAUDE.md add tokens without adding signal. The fabrication guard from #9 runs into exactly this wall.

Lambda-variance is the harder result for benchmark methodology. N=5 catches output-length variance, but if the underlying strategy varies run-to-run, mean output_tokens is a thinner signal than I'd been treating it as. Going to look at our raw-*.jsonl traces with that lens.

Thanks for sharing. Bookmarked the agent-tuning repo too.

@drona23 Along the same path, I did some exploration into understanding how thinking models solve problems. More or less, I asked the same model the same question and analyzed the thinking output over several iterations. You might be interested.

https://adamsohn.com/reliably-incorrect/ and https://adamsohn.com/lambda-variance/

Quick update tied to this thread: just merged #10 (reproducible benchmark harness) and re-ran on the current minimal `CLAUDE.md`.

`benchmark/run.py` now drives `claude -p --output-format json` in isolated temp dirs across haiku/sonnet/opus at N=5, capturing real `output_tokens`. On the current minimal profile the measured output-token reduction is roughly haiku -2%, sonnet -11%, opus -7% (excluding T4). With `profiles/CLAUDE.compressed.md` (also shipped today), opus hits -62%.

This doesn't replicate your 30-run cost-to-green coding harness, so leaving the issue open. The harness here is the single-shot Q&A side; yours covers the agentic side. Linking them as complementary.

- Fresh per-model reports: `benchmark/report-{haiku,sonnet,opus}.md`
- Token summary: `benchmark/SUMMARY.md`
- Raw per-run records: `benchmark/raw-*.jsonl`

Thanks for building this. Genuinely useful harness.

We iterated through v2-v8 using your benchmark. The original 61-line file (F-drona23) was bloated, you were right about that. We trimmed it down to 7 lines across 2 files.

Ran v8 head-to-head against C-structured on the same harness, same day, same model:

| Challenge | M-drona23-v8 | C-structured |
|-----------|-------------|-------------|
| CSV Reporter | $0.244 | $0.282 |
| SQLite Windows | $0.406 | $0.376 |
| WebSocket | $0.285 | $0.473 |
| **Total** | **$0.935** | **$1.131** |

v8 wins overall by 17.4%. The WebSocket gap is the biggest because v8 has an explicit rule to use a Set for client tracking instead of pub/sub, which prevents the debugging loop that costs extra turns.

Key takeaway we learned from your data: cost scales with turns, not file size. Each turn adds ~40k cache_read tokens. The one-shot strategy (read everything, write complete solution, test once) matters more than what's in the CLAUDE.md.

Root CLAUDE.md is...