I open-sourced a privacy-safe benchmark for coding-agent token experiments

“Total tokens” is often treated as real consumption, even though cached input, fresh input, and output have very different costs. I built agent-token-bench , a free MIT-licensed CLI that analyzes local Codex rollout files and emits numeric metrics only—never prompts, responses, reasoning content, tool arguments/results, or workspace paths. It measures: Cached versus uncached input Output and reasoning-output totals Credit-equivalent usage using a dated rate card Tool calls, polling, and compactions Per-turn and per-non-poll-action efficiency Before/after experiment changes The repository includes an anonymized initial observation, but it is explicitly described as directional—not a formal A/B result. I’d especially appreciate feedback on: The accounting and normalization methodology Adding Claude Code support Safe on/off control and intervention switching Testing Ponytail as a distinct solution-minimalism arm Which quality outcomes should be required before claiming savings