Gemini is not fast

Gemini defenders like to point at the tokens per second number. "Maybe it's not as smart, but at least it's 5x faster."

Gemini 3.8 Flash takes many, many unnecessary actions to complete a task. It can generate tokens quickly and still leave me waiting longer for the work to get done.

I test different models on the same tasks. Gemini often ignores instructions or fails things that GPT-5.6 Terra can handle, even though Gemini supposedly blows it out of the water on the benchmarks. Sometimes it decides the task is impossible, and when I call it out, it makes another attempt that goes nowhere.

Gemini 4 Argon changes some of this. From the published results, Google seems to have a legitimately okay model now. Google is starting with trusted cyber defenders, with broader access coming later. I don't have access to it yet, so the hands-on complaints here are about the earlier models. When Gemini 4 becomes available to me, some of the math may change. The way to judge it won't.

Tokens per second

The Primeagen put it pretty well: "I must admit, it's really fast at reading the same file over and over again."

Gemini loves to read files 60 lines at a time, often in overlapping chunks. If the model keeps doing that, a high tokens-per-second number doesn't help much. I want it to finish the task.

Here are the Artificial Analysis charts from September 14th, which show why tokens per second doesn’t mean faster.

Astra on low took less than half the time, cost less, and scored more than twice as high as Gemini. Sol on medium was cheaper and faster too, though it scored lower. That is a tradeoff you can choose. "Gemini is 5x faster" doesn't describe this comparison.

Artificial Analysis calculates that time from output token counts and generation speed. It excludes time to the first token and other overhead, so it isn't a stopwatch measurement of the whole job. Even on that narrower measure, Gemini's fast token generation didn't make it the fastest model per task.

More steps

Google has been improving Gemini's coding performance while making it take more steps to get there. The DeepSWE leaderboard gives a good example. Gemini 3.8 Flash on high and GPT-6 Astra on xhigh both score 74%, but Gemini averages 166 agent steps per task. Astra averages 29.

Gemini is cheaper on that benchmark, $2.36 per task against $4.43. That's a legitimate reason to choose it if you care more about cost than the extra steps. It also shows why you need to look at the benchmark you're using. The cost comparison is very different from Terminal-Bench.

An agent step isn't a fixed unit of time. Some steps are useful, and harder tasks can need more of them. But repeated file reads and attempts that go nowhere match what makes Gemini frustrating to use.

Logan Kilpatrick confirmed that this was a design decision for 3.8, while saying that measuring steps and cost was top of mind.

I want to see that curve go the other way with Gemini 4. More intelligence is good. Getting the work done with fewer unnecessary actions would be better.

The other defenses of Gemini

"But Google has so much data. They have all of YouTube. They're definitely going to dominate when it comes to video."

Then explain why ByteDance is eating their lunch. This Gemini Omni 1.1 Flash and Seedance 2.5 comparison uses the same prompt. It's one example, but owning YouTube doesn't settle which model makes the better video.

"But Google has the full stack. They make their own chips."

That could be an advantage for Google. I still have to choose a model based on what it does, how long it takes, and what it costs me. Owning more of the infrastructure doesn't answer those questions.

"But Google optimizes for normal people, not developers."

The non-coding comparisons from September didn't rescue that argument either. Gemini 3.8 Flash trailed the leading OpenAI and Anthropic models across the legal, economics, healthcare, strategy, and finance indexes in those charts. Those rankings can change, but you can't explain away every poor result by saying the test is for developers.

Gemini does have good vision. All models have strengths and weaknesses, and there are tasks where Gemini can win. If it wins at your task, use it. That doesn't make its weaknesses disappear.

Gemini 4 Argon

The Gemini 4 results look better. Depending on which benchmarks you put in your mix, it still isn't on the Pareto frontier for price versus performance. That means there are other choices that score at least as well for less money. Change the mix and you can change the answer, which is why I care about being able to choose the benchmarks.

I haven't used Argon yet. When I can, I want to run it on the work the earlier models struggled with and see whether it finishes without all the unnecessary actions.

If Gemini 4 changes the results, I'll change my recommendation. I still want to know how long the task takes, how much the whole task costs, and whether the model did it correctly. Tokens per second doesn't tell me that.