Alibaba’s new Qwen3 Max model is here, and it follows a familiar pattern for their large language models (LLMs). On paper, it looks good. It scores well on benchmarks. But when you use it in the real world, it falls short. This is not a new story for Alibaba’s LLMs; many are “sloptimized for benchmarks”—meaning they are built to ace standardized tests but struggle with actual utility or reliability in demanding tasks.
Benchmark Hype vs. Real-World Headaches: The Qwen3 Max Story
Qwen3 Max, like its predecessors, might impress on an intelligence index or other standardized tests. However, these scores often don’t translate into practical effectiveness. This is a common issue with LLMs that prioritize benchmark optimization over real-world deployment needs. My experience has been that many models, especially those from certain vendors, are more about marketing numbers than true capability. In hands-on tests, Qwen3 Max is, frankly, pretty bad. It might be a slight step up from other Alibaba LLMs, but it is not a model I would recommend for serious work.
It is important to remember that the “cost to run” an AI model, rather than just token pricing, gives a more accurate picture of its actual expense. This is because it accounts for the total tokens required to complete a task efficiently. For example, a model might have low per-token pricing but require many more tokens to achieve the desired output, making its effective cost much higher. The Artificial Analysis Intelligence Index, while a good benchmark, also comes with a high cost to run even for cheaper models due to the sheer volume of processing involved. It also does not represent every aspect of a model’s intelligence. For example, Claude Sonnet 4 scores only a 44 on the index, but it is widely considered one of the best models for coding and agentic tasks. This highlights why raw benchmark scores need to be viewed alongside practical performance and specific use cases.
This is where understanding model cost and true performance becomes critical. Organizations need models that balance benchmark performance with real-world usability and cost-effectiveness. Open-source models that can run on various hardware platforms, like Cerebras, and those with proven strengths in specific domains, such as coding, are gaining traction over models that only perform well on benchmarks alone. There is a recognition in the industry that benchmark optimization is not enough. Models must demonstrate reliability, versatility, and genuine value in practical deployments to achieve widespread adoption.
The Qwen3 Coder Anomaly: A Claude Distill
The one major exception in Alibaba’s LLM lineup is Qwen3 Coder. This model is genuinely strong, particularly in coding and agentic tasks. Its quality is probably due to training on outputs from Anthropic’s Claude, which is a leading model known for its excellence in these areas. Qwen3 Coder’s open-source nature means you can run it with any provider, including Cerebras, which makes it incredibly fast and cost-effective. This unique combination of performance, accessibility, and speed has allowed it to gain significant market share among developers who need powerful, budget-friendly coding models.
Qwen3 Coder is the only Alibaba LLM to achieve meaningful adoption. Its success isn’t from internal innovation in general-purpose intelligence, but rather from a clever “Claude distill”—a model trained by distilling the outputs of a superior model. This shows that while open-source models can sometimes leapfrog to the frontier, proprietary companies often re-incorporate these advancements into their offerings. As I have said before, open source models are often a couple of months behind closed source models but are valuable for privacy and cost reduction.
| Model | Type | Archetype | Strengths | Weaknesses |
|---|---|---|---|---|
| Claude Sonnet 4 | Hybrid | Artisan | Great at coding, writing, design; good for tools/agents | Weak on heavy reasoning, expensive |
| Qwen3 Coder | Non-reasoning | Apprentice | Great at coding, blazing fast (Cerebras), good tool calling, front-end design | Weak on creativity, reasoning; no Vision |
| GPT 5 Thinking | Reasoning | King | Best world knowledge, highest innate intelligence, great at writing, very creative, great at code | Not reliable in coding agents w/o special prompting, overkill for many tasks, weaker tool calls vs Claude |
A comparison of Qwen3 Coder with other leading models, highlighting its unique strengths.
Beyond Text: Alibaba’s Strength in Multimodal Models
While Alibaba’s text-based LLMs have generally been underwhelming (with Qwen3 Coder being the obvious outlier), the company has found success elsewhere. They have produced notable open-source image and video generation models, such as Qwen Image and Wan 2.2. These models have been well-received for their quality in generative tasks that extend beyond text. This suggests that Alibaba’s AI capabilities are strong, but their strategic focus for LLMs appears to be heavily skewed towards benchmark performance rather than general utility.
It is worth noting that while models like Qwen3 Max are “sloptimized” for benchmarks, this is not an uncommon practice in the industry. Many labs optimize for specific, well-known benchmarks that might not capture the full range of abilities of a “smart” model. For instance, models that score highly on specific reasoning tests might struggle with complex, multi-step agentic workflows that require persistence and nuanced understanding, as opposed to simply answering questions correctly. This is one of the limitations of relying solely on an “Intelligence Index” for model selection.
The Intelligence Index: A Tool, Not the Whole Story
The Artificial Analysis v3.0 (Sept 2025) Intelligence Index, as described, blends eight text-only evaluations to compare models across reasoning, knowledge, math, coding, instruction following, long-context use, and agentic workflows. It includes metrics like MMLU-Pro, HLE, GPQA Diamond, AIME 2025, IFBench, SciCode, LiveCodeBench, AA-LCR, plus two agentic sets, Terminal-Bench Hard and τ²-Bench Telecom. While it offers a standardized way to compare models, it doesn’t always tell the whole story. A model like Claude Sonnet 4 (Intelligence Index: 44) can still be a top performer in specific domains like coding and agentic tasks, where it truly excels on lesser-known benchmarks that labs don’t optimize for as much. This is because raw scores don’t capture every nuance of a model’s intelligence or its suitability for particular applications. This is why I caution against making sweeping claims about one model being universally “better” than another based solely on its Intelligence Index score. The broader model analysis, which details strengths and weaknesses, provides a more reliable guide for performance.
The methodology for the Intelligence Index is rigorous, using standardized prompts, zero-shot instruction style, transparent answer extraction, and grading that minimizes unfair penalties for valid formats. It uses multiple choice with strict “Answer: X” formats and regex fallbacks, and equality-checker LLMs for open answers. Graders include GPT-4o Aug for HLE correctness, Llama 3.3 70B for math equivalence, and Qwen3 235B non-reasoning as an equality checker for long-context tasks. Agentic runs have extensive timeouts and repetition checkers to prevent loops. Testing defaults include temperature 0 for non-reasoning models and 0.6 for reasoning models, generous output token caps, and automatic API retries. The weighting balances general knowledge, reasoning, math, coding, instruction following, and long-context reasoning, with V3.0 adding agentic benchmarks. The purpose is to provide a broad, zero-shot, text-only benchmark that mixes academic Q&A, competition math, program synthesis, long-document reasoning, and simulated agent tasks, with explicit, published prompts and extraction rules to keep runs consistent and comparable. It’s a robust system, but still a system with a specific scope.
| Model | Intelligence Index | Cost to Run ($) | Primary Strength (if notable) |
|---|---|---|---|
| GPT-5 (high) | 67 | 873 | Highest innate intelligence, world knowledge |
| Grok 4 | 65 | 1764 | Best for long running agents, great at math/code |
| Claude 4.1 Opus Thinking | 59 | 2335 | Great at coding, writing, design, tools/agents |
| Qwen3 235B 2507 (Reasoning) | 57 | 910 | Good for chat, follows niche data structures well |
| Claude 4 Sonnet Thinking | 57 | 665 | Great at coding, writing, design, good for tools/agents |
| Qwen3 Coder 480B | 42 | 80 | Great at coding, blazing fast (Cerebras), good tool calling |
Selected AI models compared by their Intelligence Index and Cost to Run, illustrating the balance between reported intelligence and operational expense.
The Broader Trend: Beyond Benchmark Obsession
The industry’s focus is shifting. Raw benchmark scores alone are no longer sufficient. Models need to demonstrate reliability, versatility, and genuine value in practical deployments to gain widespread adoption. This means looking beyond an impressive Intelligence Index score to understand a model’s true capabilities and limitations. For me, this often comes down to specific use cases. Is the model good for coding? Is it good for agentic workflows? Is it cost-effective for high-volume tasks? These practical questions often matter more than a general intelligence score.
Models like OpenAI’s GPT-OSS-120B are open-source and can run extremely fast on Cerebras, making them useful for quickly formatting large amounts of data, despite not being top performers in general intelligence. Similarly, models like GPT-4.1 are designed for developers with strengths in reliable tool calling and structured output, catering to specific needs rather than attempting to be universally “smart.” Even smaller specialized models like GPT-5 Nano offer strengths in summarization and structured output at an extraordinarily cheap rate, highlighting specialized value over raw intelligence.
It’s an important distinction: a model optimized for coding, like Qwen3 Coder, can be functionally superior for that specific task, even if its overall intelligence index is lower than a general-purpose model. This is where models like Moonshot’s Kimi K2.1 or Claude for Chrome fit in. They are tailored for specific functions and excel within those boundaries, demonstrating that true utility often comes from specialized competence rather than brute-force general intelligence.
The key takeaway is that when considering an LLM, particularly open-source ones, simply looking at benchmark numbers can be misleading. A model’s real value lies in its fit for your specific tasks, its actual cost-efficiency, and its reliability in deployment. This is why models like Qwen3 Coder, which likely benefited from distilling the intelligence of a frontier model like Claude, stand out.
This situation also reminds me of my thoughts on OpenAI’s Codex IDE extension, where aggressive moves are made to control market share. Alibaba’s success with Qwen3 Coder, essentially a ‘Claude distill’ that found product-market fit due to its open-source nature and performance on platforms like Cerebras, shows that even companies with less successful general LLMs can carve out a valuable niche through strategic approaches to model development and distribution.
Conclusion: Max’s Minimal Impact and Coder’s Success
Qwen3 Max is another example of Alibaba’s pattern of producing LLMs that appear strong on benchmarks but do not perform well in practice. For most real-world applications, Qwen3 Max and its general-purpose siblings are not models I would recommend. Alibaba’s only true LLM success has been Qwen3 Coder, and its effectiveness directly stems from distillation from Anthropic’s Claude and its flexible, open-source deployment options that allow for incredible speed on platforms like Cerebras. In the world of LLMs, practical utility and specific domain strength often trump abstract benchmark scores, and Qwen3 Coder is a clear demonstration of this principle.