Cost Per Correct is a five-task, easy-tier benchmark that grades LLM agents on cost per correct answer — total spend (USD) ÷ number of correct answers — instead of accuracy alone, because a cheap model that is usually right can beat an expensive model that is always right in production. Each task is a short, single-step agent-style job (arithmetic, extraction, classification) graded by a deterministic assertion, with no LLM judge, and run cost is accounted with tokenecon. This page is the living leaderboard: it is regenerated from the hosted results in the dataset repo, so verified community submissions added there appear here.
Links: Kaggle benchmark (canonical live run surface) · Hugging Face dataset · Zenodo archival record — DOI 10.5281/zenodo.23439614 · Cost model preprint: Token-Economics for Agentic AI (DOI 10.5281/zenodo.23195511)
Notes on reading the table: “attempted” counts tasks with a registered model result (Pass or Fail). Kaggle platform errors and missing runs are disclosed in the Notes column and excluded — they are not model answers. Kaggle reports pass/fail only, so cost per correct is shown where a priced run reports it (starting with submissions that include spend; see below).
Data source: live from the dataset repo (https://huggingface.co/datasets/karmendra8386/cost-per-correct). 5 leaderboard row(s).
| Model | Tasks passed / attempted | Accuracy % | Cost per correct (USD) | Date | Notes |
|---|---|---|---|---|---|
| Gemini 2.5 Flash | 5 / 5 | 100.0 | — | 2026-10-11 | |
| DeepSeek R1 | 3 / 3 | 100.0 | — | 2026-10-11 | Platform error (not a model result): e2; No registered result: e1 |
| Gemini 3.7 Flash | 3 / 3 | 100.0 | — | 2026-10-11 | No registered result: e1, e2 |
| Qwen3 235B | 3 / 3 | 100.0 | — | 2026-10-11 | Platform error (not a model result): e2; No registered result: e1 |
| GPT-OSS 120B | 2 / 3 | 66.7 | — | 2026-10-11 | Platform error (not a model result): e2; No registered result: e1 |
Last updated October 19, 2026 — regenerated whenever a verified result is merged into the dataset.
| Task | Gemini 2.5 Flash | Gemini 3.7 Flash | DeepSeek R1 | GPT-OSS 120B | Qwen3 235B | Note |
|---|---|---|---|---|---|---|
| e1 | Pass | no result | no result | no result | no result | Only Gemini 2.5 Flash has a registered result on e1. |
| e2 | Pass | no result | Error | Error | Error | The three Error rows are Kaggle platform errors: the runs completed but the platform would not register swapped-in models. Not model answers. Disclosed in the entry write-up (dev.to, 2026-10-11). |
| e3 | Pass | Pass | Pass | Pass | Pass | |
| e4 | Pass | Pass | Pass | Pass | Pass | |
| e5 | Pass | Pass | Pass | Fail | Pass | GPT-OSS 120B failed this extraction task; all other models passed. |
Anyone can run the benchmark and submit results. Submissions are reviewed and, once verified, added to the dataset — and therefore to this leaderboard.
1. Get the tasks. Download cost-per-correct-tasks.json from the
dataset repo. It contains the 5 prompts, the expected answers,
and the deterministic grading assertion (a regex) for each task.
2. Run the tasks. Send each prompt to your model once, end-to-end, the way an agent would receive it. Grade each response with the task’s regex assertion — a response passes or it does not; there is no LLM judge. If you want a cost-per-correct figure, price the run with tokenecon and record the total spend in USD and the provider price list you used.
3. Prepare your results CSV — the same schema as hosted-results-2026-10-11.csv:
task_id,<your_model_id>,note
e1,Pass,
e2,Pass,
e3,Fail,
e4,Pass,
e5,Pass,llama_3_3_70b).Pass, Fail, or — only for infrastructure
failures that are not model answers — Error, with the reason in the
note column for that row.4. Open a Discussion (or a Pull Request) on the dataset repo:
karmendra8386/cost-per-correct — Discussions, titled
Result submission: <model name> (<run date>), and include:
Verified submissions are merged into the dataset repo and appear in the leaderboard above with their run date. Questions: open a Discussion on the dataset repo.