Heres the state of LLMs today: Artificial Analysis Intelligence Index v3.0 is the scoreboard for September. The market sorts into three tiers that actually matter when youre building: Great models for 80% of real work, Decent models for the remaining 20%, and a set you should ignore. Use v3s scores to pick the tier, then decide on price and latency. Thats the whole play.
Artificial Analysis Intelligence Index v3.0: what the scores mean
At the start of the month, Artificial Analysis shipped Intelligence Index v3.0, a broad, zero-shot, text-only benchmark that blends eight core Q&A and coding tasks plus two agent testbeds. The suite covers MMLU-Pro, HLE, GPQA Diamond, AIME 2025, IFBench, SciCode, LiveCodeBench, and AA-LCR, with two agentic sets added in v3: Terminal-Bench Hard and cccccccc 7f tcc 7f 7f 7f 7f 7f. Scoring uses standardized prompts, strict extraction rules (Answer: X style for MCQ with layered regex fallbacks), and equality-checkers for open answers. Grading LLMs are disclosed, including GPT-4o Aug for HLE checks, a Llama 3.3 70B plus SymPy combo for math equivalence, and Qwen3 235B non-reasoning for long-context equality. Agent runs cap episodes at 100 with a repetition checker after 30, and long timeouts to prevent loops. They report b1 1% uncertainty on the overall Index, which is tight enough to treat close scores as ties in practice.
Defaults matter: temperature 0 on non-reasoning models, 0.6 on reasoning models, generous output caps, Ubuntu 22.04 plus Python 3.12 for code tasks, plus robust retry logic with manual review on persistent failures. In short, v3.0 is a transparent, reproducible way to compare a lot of models under realistic, zero-shot instructions. Use it for portfolio decisions, not just leaderboard screenshots.
Closed vs open: who sits where
Closed models still hold the highest raw intelligence and the best generalist performance. Open models push cost down, throughput up, and give you more deployment options, especially with providers like Cerebras or Groq. If youre chasing the absolute best reasoning or agent reliability, youre usually in closed territory. If youre building high-volume workflows and careful about scope, open models can be a price and performance win.
The 80-20-0 rule
Heres the split I use and whats implied by the v3 runs:
- Great models (80% of picks): GPT-5 Thinking, Claude Sonnet 4, Qwen3 Coder, GLM 4.5, Gemini 2.5 Pro. If your work matters, start here and prune based on cost and latency.
- Decent models (20%): The rest of the strong, practical options for cost-sensitive workloads, batch transforms, and tasks that dont need the absolute best intelligence.
- Trash models (0%): Dont bother. Examples not even included in the main data: Llama 4 Maverick, Scout, Llama 3.3 70B, Minimax M1 Destral, Magistral Small, Nemotron.
Top five Great cards
- GPT-5 (high) Intelligence 67, cost to run $873. Strengths: best world knowledge, top-tier reasoning, excellent writing and code. Weaknesses: overkill for many jobs, tool use not as smooth as Claude on complex stacks.
- Claude 4 Sonnet Thinking Intelligence 57, cost to run $665. Strengths: coding, writing, design, agent loops with tools. Weaknesses: weaker on very heavy reasoning vs top frontier peers, pricey.
- Qwen3 Coder Intelligence 42 and cost $80 for the 480B route in our data. Strengths: code throughput and reliability for structured tasks, great tool-calling. Weaknesses: creativity, deep reasoning, no vision.
- GLM-4.5 Intelligence 49, cost $232. Strengths: good coding and writing, solid for tools and agents, inexpensive. Weaknesses: heavy reasoning trails frontier, no vision.
- Gemini 2.5 Pro Intelligence 60, cost $1012. Strengths: very smart, strong code and design, long context, multimodal. Weaknesses: reliability and steer-ability can be shaky, tool use inconsistent.
These five are where serious work gets done. You will pay more for the top two, and youll usually get shorter paths to working outputs and fewer retries. On longer runs, that matters more than the sticker price.
Price vs intelligence
- Grok 3 mini Reasoning (high) lands at Intelligence 57 with a $57 run. Thats an obvious pick for cost-constrained agents where trace depth is helpful.
- Claude 4.1 Opus Thinking at Intelligence 59 and $2335 is very capable but not a volume choice.
- Grok 4 is smart (65) and expensive ($1764). If you keep it on short bursts with sharp prompts, it can make sense. For long-running loops, budget first.
- GLM-4.5, DeepSeek V3.1 (Reasoning), and Kimi K2 0905 form a solid mid-cost band that covers many daily agent needs.
The 20 cards
The middle class is crowded, and thats good. Useful picks include:
- o3 Intelligence 65, cost $407. Reliable reasoning and strong tool use. Replaced by GPT-5 Thinking for top-end jobs, but still a steady worker.
- o4-mini (high) Intelligence 59, cost $337. Fast, cheap, and reliable reasoning. Weak at creative writing. Good for complex agents that do not need prose.
- GPT-5 (medium) Intelligence 66, cost $460. A ton of capability without the full price of the high tier.
- GPT-5 mini (medium and high) Intelligence 61 at $57, or 62 at $170. Great structured outputs, strong for cheap agents and research helpers.
- Gemini 2.5 Flash (Reasoning) Intelligence 51, cost $235. Long context, multimodal IO, budget friendly.
- Kimi K2 0905 (K2.1) Intelligence 51, example run $78. Very fast on Groq, strong tool calling for its weight, but watch hallucinations.
- DeepSeek V3.1 (Reasoning) Intelligence 54, cost $141. Good open pick when you want low-mid cost with decent reasoning traces.
- Qwen3-Next-80B-A3B Intelligence 45 (route dependent), example run $92. Two routes (Instruct vs Thinking). Cheap and large context, but validate per route before production. I covered the tradeoffs here: Qwen3-Next-80B-A3B: Cheap But Test Before You Commit.
- GLM-4.5 see above. If you are wiring tools and do not need frontier logic, this is in the rotation.
Cost-to-run shortcuts
- Under $20: GPT-4.1 mini ($17, Intelligence 42), DeepSeek V3.1 Non-reasoning ($17, 45). Bulk transforms, chat support, simple extractors.
- Under $60: GPT-5 nano medium ($25, 48), GPT-OSS-20B high ($11, 45), GPT-4.1 nano ($3, 27). High-volume formatting and summarization. Also look at Grok 3 mini Reasoning (high) at $57 for agent loops with traces.
- $60350: GLM-4.5 ($232, 49), DeepSeek V3.1 Reasoning ($141, 54), Kimi K2 0905 ($78, 51), Qwen3 Coder 480B ($80, 42). Good for mid-scale coding and agents.
- $4003000: GPT-5 high ($873, 67), Gemini 2.5 Pro ($1012, 60), Claude 4 Sonnet Thinking ($665, 57), o3 ($407, 65). These are the flagship picks for high-stakes runs.
Pricing details on some routes: Kimi K2 0905 is often served on Groq, with $1.00 in and $3.00 out in some listings (native published around $0.60 in and $2.50 out). Qwen3 Max has an estimate placeholder in some routes ($1.20 in and $6.00 out). Always check your providers live rates and batch settings.
Agentic tests in v3 and what I am watching
v3 adds Terminal-Bench Hard and tcc-Bench Telecom for agents. The runs cap episodes at 100 and trip a repetition checker after 30 to avoid loops. This aligns with the reality of long-running coding or research agents. You pay for planning tokens and retries, not just correct final answers. That is why reliability beats raw creativity for agents. If your agent keeps its head, you save budget.
Related reading on model behavior and calibration: ConfidenceBench: Calibrating LLM Confidence, Not Just Accuracy.
Practical picks by job
- Deep research, long reports, hard synthesis: GPT-5 (high or medium). If cost is tight but you still need traces, consider Grok 3 mini Reasoning at $57 with Intelligence 57. Gemini 2.5 Pro is an option for very long context and multimodal review, but steer-ability can cost you retries.
- Day-to-day coding agent: Claude 4 Sonnet Thinking and o3 are safe defaults when tool-calling matters. Grok Code Fast 1 is better than I expected for this lane; I wrote about that here: Grok Code Fast 1: I Was Wrong.
- Cheap agents at scale: GPT-5 mini (medium or high), GLM-4.5, Kimi K2 0905. Validate hallucination rates before you roll out K2.1 broadly.
- Bulk formatting and summarization: GPT-5 nano (high and medium), GPT-4.1 nano, GPT-OSS-120B on Cerebras. Throughput is the whole story here.
- Long-context reading: Gemini 2.5 Pro and GPT-4.1 both work well when you need to shove in a lot of text and keep extraction predictable.
- Front-end code and UX content: GPT-5 Thinking and Claude 4 Sonnet deliver cleaner prose and UI polish than most mid-tier models. If cost forces you lower, accept you will rework more outputs.
What to skip
Do not chase models that do not show up in the reliable data. The trash list is not subtle: Llama 4 Maverick, Scout, Llama 3.3 70B, Minimax M1 Destral, Magistral Small, Nemotron. Also treat Qwen3 Max as sloptimized until proven otherwise. Pricing looks fine on paper, but tests show it overfits patterns and fumbles real coding work. If you must try it, fence it inside narrow tasks.
Notes on tool use and naming chaos
Tooling is where Claude Sonnet 4 shines. It is not the strongest at maximum-depth reasoning, but if your agent calls multiple tools, Sonnets consistency pays for itself in reduced retries and cleaner planning. GPT-5 Thinking is the raw intelligence leader and can do everything, but you may need prompt scaffolds for complex toolplay. Gemini 2.5 Pro can be excellent on long contexts yet fussy to steer. It is workable as long as you budget for a few misfires.
Model naming continues to be messy across vendors. If you are mapping products to engines for coding agents, this piece is relevant: OpenAI Codex IDE Extension: When AI Coding Meets Confusing Product Names. Also note my breakdown on how OpenAI is routing models inside Codex here: GPT-5-Codex inside Codex.
Cost-to-run snapshots from this months data
- OpenAI: GPT-5 (high 67, $873), GPT-5 (medium 66, $460), o3 (65, $407), o4-mini high (59, $337), GPT-5 mini high (62, $170), GPT-5 mini medium (61, $57), GPT-4.1 (43, $69), GPT-4.1 mini (42, $17), GPT-4.1 nano (27, $3), GPT-5 nano medium (48, $25), GPT-5 nano high (49, $61), GPT-5 minimal (43, $50).
- Anthropic: Claude 4.1 Opus Thinking (59, $2335), Claude 4 Sonnet Thinking (57, $665), Claude 4 Sonnet (44, $126).
- Google: Gemini 2.5 Pro (60, $1012), Gemini 2.5 Flash Reasoning (51, $235), Gemini 2.5 Flash (40, $53), Gemini 2.5 Flash-Lite Reasoning route (40, $57) and Lite (30, $20).
- xAI: Grok 4 (65, $1764), Grok 3 mini Reasoning high (57, $57), Grok Code Fast 1 (49, $132).
- Open-weight and others: Qwen3 235B Reasoning (57, $910), Qwen3 235B Non-reasoning (45, $71), Qwen3 Coder 480B (42, $80), Qwen3 Coder 30B (33, $33), Qwen3-Next-80B-A3B (45, $92 example run), Qwen3 Max (44, $120 placeholder), GLM-4.5 (49, $232), DeepSeek V3.1 Reasoning (54, $141), DeepSeek V3.1 Non-reasoning (45, $17), Kimi K2 (46, $78), Kimi K2 0905 K2.1 (51, $78 example run), GPT-OSS-120B high (58, $69), GPT-OSS-20B high (45, $11).
Bottom line
Pick your tier first. If the work is important, the Great bucket saves time and reduces retries. If the job is narrow and price sensitive, the Decent bucket has plenty of quality. The trash bucket is noise. Intelligence Index v3.0 is the best single reference for making those calls right now. Match the model to the job, measure cost-to-run under your prompts, and do not scale anything that you have not validated on your actual workload.
Further reading relevant to this months picks: Grok Code Fast 1: I Was Wrong Qwen3-Next-80B-A3B: Cheap But Test Before You Commit ConfidenceBench: Calibrating LLM Confidence GPT-5-Codex inside Codex.
Data reflects test settings and routes noted. Some routes list example runs or placeholders that vendors may update. Always recheck your provider for live pricing and context limits before scaling.