ROIpad ← Back to Search
github.com › issue comment

Comment on: Empirical evaluation: 30-run benchmark of token-efficient vs 4 other .claude/ configs

Repo: drona23/claude-token-efficient by drona23
Posted: Jun 15, 2026
Quick update tied to this thread: just merged #10 (reproducible benchmark harness) and re-ran on the current minimal `CLAUDE.md`. `benchmark/run.py` now drives `claude -p --output-format json` in isolated temp dirs across haiku/sonnet/opus at N=5, capturing real `output_tokens`. On the current minimal profile the measured output-token reduction is roughly haiku -2%, sonnet -11%, opus -7% (excluding T4). With `profiles/CLAUDE.compressed.md` (also shipped today), opus hits -62%. This doesn't replicate your 30-run cost-to-green coding harness, so leaving the issue open. The harness here is the single-shot Q&A side; yours covers the agentic side. Linking them as complementary. - Fresh per-model reports: `benchmark/report-{haiku,sonnet,opus}.md` - Token summary: `benchmark/SUMMARY.md` - Raw per-run records: `benchmark/raw-*.jsonl`