Comment on: Empirical evaluation: 30-run benchmark of token-efficient vs 4 other .claude/ configs
Repo: drona23/claude-token-efficient by drona23
Read both. The p_step^N framework lines up with what our N=5 semantic eval showed.
In SEMANTIC.md from #10, judge accuracy and completeness are 100% in both conditions across haiku, sonnet, opus. Em-dash and verbosity markers move; correctness does not. The rules change tone, not capability. That matches your point: p_step is set by what's dense in training, not by the system prompt.
The verification-on-same-weights observation is the cleanest framing I've seen for why "check your work" rules in CLAUDE.md add tokens without adding signal. The fabrication guard from #9 runs into exactly this wall.
Lambda-variance is the harder result for benchmark methodology. N=5 catches output-length variance, but if the underlying strategy varies run-to-run, mean output_tokens is a thinner signal than I'd been treating it as. Going to look at our raw-*.jsonl traces with that lens.
Thanks for sharing. Bookmarked the agent-tuning repo too.
GitHub Issue
Parent Entity
State: Open • Comments: 1
SaaS Metrics