From the pack / Product notes
Dogbench: measure runs, not promises
A smaller prompt is an observation. A cheaper or better development workflow is a separate claim. Dogbench exists to explore those measurements.
Compare complete runs
The Dogbench repository describes a local harness for collecting usage from available agent clients and comparing tasks. Treat it as experimental tooling, not a public leaderboard or proof that one harness always wins.
For a useful comparison, pin the task and input revision, model/provider, runtime, permissions and cache state. Include retries, tool traffic and output tokens rather than reporting only the initial context. Mark unavailable usage as unavailable; do not turn it into a zero.
Correctness comes before a savings percentage
A fast result that omits the relevant relationship or fails the task is not an improvement. Check the returned facts or task outcome, repeat runs and retain the variance. Do not mix tokenizer counts, byte-based estimates and provider billing units.
The Dotdog release comparison follows a narrower method: same-source builds, stated warm-cache conditions, repeated workflow timings and unchanged-result checks. It does not establish coding-agent productivity or general cloud cost savings.
No synthetic scoreboards
We are not publishing sample dollar amounts as live customer results. Dogbench's older README examples are illustrative and are not used as evidence for website claims. Its full current comparison/provider surface has not been independently reproduced by this website update.
Before sharing a run, review outputs for prompts, local paths, account identifiers and credentials. Prefer a sanitized report with the exact scope over raw session logs.