Why we run Gemini 3.7 Flash on medium, not high

Google shipped Gemini 3.7 Flash yesterday. I had it running on GrandpaCAD within the hour, and the switch itself was boring: cheaper than what we ran before, fast, and it lands a printable model on the first try. The interesting part was a small experiment I almost skipped, about how hard to let the model think.

Short version: more reasoning fixed a problem we'd already solved, and charged us for the privilege. We kept it on medium.

The switch

3.7 Flash is now the default for both the legacy generator and the agent chat. Intro pricing is $0.75 per million input tokens and $3.75 per million output, well under the frontier models we've been leaning on. For a text-to-3D workload where people iterate (prompt, look, tweak, prompt again), a cheap fast model that lands a valid part beats a slow expensive one that's marginally prettier. Same argument I made when I told you to test on your own workload instead of public leaderboards, just pointed at a new model.

The thinking experiment

Gemini lets you dial reasoning effort: low, medium, high. I ran our standard eval suite twice, once on medium and once on high, everything else identical. 23 prompts, one attempt each.

High thinking did exactly what you'd hope on one axis. First-try code errors dropped from 10 to 3. The messy multi-turn prompts that usually trip a model up (a phone stand that limped through three broken attempts on medium) came out clean on the first pass.

And then it lost anyway.

Why fewer errors didn't win

Here's the thing about those code errors: the user never sees them. When generated code throws, the agent feeds the error back to the model and it retries inside the same turn, before anything reaches your screen. Both runs produced a finished model for all 23 prompts. So "fewer first-try errors" is a metric that mostly counts work happening backstage.

What reaches you is the finished part, the wait, and (on our side) the bill. On all three, high thinking was worse. It cost about 35% more per model and took about 15% longer. And when I graded the results by hand, the high-thinking parts came out slightly lower on adherence, not higher. At this sample size I'd call adherence a wash, but that's the point: it wasn't better. I paid extra for more thinking and got the same models, slower.

(Ignore any "high scored lower on checks" claim you might infer. The only checks it failed were a 60-second timer, blown by the longer thinking. None were about correctness.)

The actual lesson

I keep relearning this one. A few months back the Gemini 3.1 upgrade taught me that medium thinking beat high thinking on error rate. This time high thinking won on error rate and still lost. Opposite mechanism, same conclusion: the intermediate metric lied about the product.

If you're tuning a model for something real, measure the thing the user actually gets. Not the retries, not the token count, not the reasoning trace. For us that's a printable model, how long you waited, and how close it landed. First-try error rate felt like it should matter. For a pipeline that already cleans up its own errors, it didn't.

The honest caveats

This was our cheap test loop: one attempt per prompt, no LLM judge, my own eyes on the adherence scoring. It's a directional read, not a verdict. One run per prompt means a few of those numbers are noise, and I won't pretend 0.80 versus 0.72 is a clean result. The full suite runs three attempts with automated judges, and if it flips any of this, I'll write it up. Every run, including the broken ones, is logged on the evals page.

For now: Gemini 3.7 Flash, medium thinking, live on GrandpaCAD.