Comment on: Empirical evaluation: 30-run benchmark of token-efficient vs 4 other .claude/ configs
Repo: drona23/claude-token-efficient by drona23
Quick update tied to this thread: just merged #10 (reproducible benchmark harness) and re-ran on the current minimal `CLAUDE.md`.
`benchmark/run.py` now drives `claude -p --output-format json` in isolated temp dirs across haiku/sonnet/opus at N=5, capturing real `output_tokens`. On the current minimal profile the measured output-token reduction is roughly haiku -2%, sonnet -11%, opus -7% (excluding T4). With `profiles/CLAUDE.compressed.md` (also shipped today), opus hits -62%.
This doesn't replicate your 30-run cost-to-green coding harness, so leaving the issue open. The harness here is the single-shot Q&A side; yours covers the agentic side. Linking them as complementary.
- Fresh per-model reports: `benchmark/report-{haiku,sonnet,opus}.md`
- Token summary: `benchmark/SUMMARY.md`
- Raw per-run records: `benchmark/raw-*.jsonl`
GitHub Issue
Parent Entity
State: Open • Comments: 1
SaaS Metrics