AI Model Pricing Is No Longer Just About the Token. Cost Per Task Changes the Math

Four model pairs, close scores, very different costs

The Artificial Analysis Intelligence Index is a composite score built from ten evaluations — AA-Briefcase, GDPval-AA v2, τ³-Banking, Terminal-Bench v2.1, SciCode, Humanity's Last Exam, GDP.pdf, CritPt, AA-Omniscience, and AA-LCR v1.1, in version v4.2 of the index — measuring reasoning, coding, knowledge, instruction following, scientific reasoning, and the ability to complete multi-step tasks. For every model, the platform separately calculates the weighted average cost, in dollars, of running one task from the index: from the model's prices for input, for cache (text already sent once and reused, billed separately), for reasoning (the tokens a model produces while thinking before it answers), and for output, divided by task count and weighted by each evaluation's share of the index. It's a far closer proxy for "what it costs to actually run a task" than the plain price per million tokens. It isn't, however, the cost of a task solved correctly: answer quality is measured separately, through the Intelligence Index score — "Cost per Task" shows what running it costs, not whether it succeeded.

Four comparisons, checked directly on artificialanalysis.ai on September 5, 2026, at the effort configuration available for each model on the public leaderboard:

The score gap inside each pair is one or two points, and in the first pair the cheaper model scores higher than the expensive one. On a composite index built from ten evaluations, one or two points don't automatically translate into a "quality" percentage for a given company's workload; the Intelligence Index is a relative score, not a linear scale of economic performance. What can be stated with certainty, because it's directly verified: cost per task differs by 2.4x to 18.1x between models in the same score band.

One point apart on score, one model produces 150 million tokens and the other 14 million

A lower cost per task doesn't automatically mean a "weaker" or simpler-to-run model, and a higher one doesn't mean a more efficient model either. The Qwen3.8 Max / GPT-5.6 Sol (medium) pair shows it most clearly: 47 against 46 on the Intelligence Index, a single point, yet Qwen3.8 Max costs $1.188 per task against $0.367, and Artificial Analysis labels it "very verbose" — 150 million output tokens to get through the full index, against a 79 million median for its class. Sol (medium) is "very concise," at 14 million tokens, more than ten times fewer. Speed runs the same way: 40 tokens per second for Qwen3.8 Max, 71 for Sol.

The pair with the widest cost gap runs the other way. Claude Sonnet 5 is the most talkative model in the table, at 320 million output tokens against a 79 million median for its class — and it scores lower than GLM-5.3-Flash, which costs 18.1x less. GLM-5.3-Flash is no token miser either: also "very verbose," at 160 million against a 140 million median for its class, plus a "notably slow" label, at 47 tokens per second against 75 for Sonnet 5.

In practice: a lower cost per task at a similar intelligence score says nothing about how fast a model responds or how much text it produces to get there — two things that matter directly in production: latency, and the volume of tokens the model will burn through in your own workflow, which can differ from the behavior observed on the benchmark.

Why $/million tokens is no longer enough

The real cost of an AI task inside a company includes the cost of the tokens actually consumed, the cost of the retries when the first answer isn't good enough, the latency that slows down an automated flow, and the human review time needed before the result can be used. None of the last three factors show up on a vendor's pricing page — and none of them fully show up in "Cost per Intelligence Index Task" either, which remains the cost of running the Artificial Analysis benchmark, not the cost of running a company's own workload.

Artificial Analysis itself acknowledges the limit: alongside the general leaderboard, the platform now offers Optima, a tool that lets any company build its own benchmark from the real tasks it runs, to see which model actually wins on its own work — not just on the index's ten generic evaluations. The practical takeaway: the Intelligence Index and cost per task show where to look; a company's own workload decides what it buys.

What a company should recalculate at every contract renewal

• Cost per completed task, at the quality bar the company accepts — not just the price per million tokens listed by the vendor.
• Verbosity and output tokens on the company's own workload, not just on the benchmark score.
• First-attempt success rate, and how often retries or corrections are needed.
• Latency and end-to-end response time, especially in flows where one step waits on another's result.
• The human review time needed before a result can be used without further checking.
• Confidentiality and data-residency requirements — a much cheaper model doesn't help if it fails the company's security policy.
• Switching and integration costs if the vendor changes — MassAI has already covered how quickly dependence on a single vendor can turn into an operational risk.

Commoditization doesn't mean every model has become the same. It means that, for a growing number of workloads, a performance difference has to be justified economically, not assumed from the model's brand. The practical decision for any company renewing an API contract: don't ask only what the million tokens cost — calculate the cost per accepted outcome instead: inference price, plus retries, plus latency, plus human review time, plus switching cost, not just the number on the pricing page. Run the same workload on 3-5 models and compare the cost of a completed task, at the quality level your company accepts — the same principle behind any well-built operational AI agent, where choosing a model is an architecture decision, not a brand reflex.

Sources: ↗ Artificial Analysis — LLM Leaderboard · ↗ Artificial Analysis — Claude Sonnet 5 · ↗ Artificial Analysis — GLM-5.3-Flash · ↗ Artificial Analysis — Qwen3.8 Max · ↗ Artificial Analysis — GPT-5.6 Sol (medium) · ↗ Artificial Analysis — Optima