Claude Opus 5 on real 3D generation
Claude Opus 5 landed, so I pointed the eval suite at it. 23 scenarios, replayed through the actual chat agent with the real CAD tools, nothing mocked. The methodology is here if you want the details.
I ended up running it three times. The first two differed only in reasoning effort. The third changed only which provider the requests went through. Both of those one-line changes moved the numbers more than picking a different model would have.
The numbers
All three of these ran through OpenRouter, so the effort comparison is apples to apples.
| Metric | Opus 5 (low) | Opus 5 (high) | Fable 5 (low) | Gemini 3.6 Flash |
|---|
| Deterministic checks | 84% | 69% | 81% | 85% |
| Code errors | 0 | 5 | 2 | 8 |
| Total cost | $13.99 | $19.39 | $12.17 | $5.23 |
| Avg duration | 116s | 275s | 94s | 107s |
High reasoning effort was strictly worse
Not "more expensive for a small gain." Worse. Low effort was 28% cheaper, 2.4x faster, and went from five code errors to zero, while the judge's adherence score barely moved (0.92 to 0.90, which is noise at this sample size).
The failure pattern explains it. At high effort, 30 of the 33 failed checks were cost or duration budget violations rather than wrong answers. The model wasn't getting the geometry wrong. It was thinking itself past the wall clock. Two multi-turn scenarios blew straight through the 10 minute per-generation timeout and got killed mid-run. At low effort, no timeouts, and only one cost failure left.
I've hit this before. Back in the GPT-5 sweeps, medium reasoning beat high for about half the cost. Same shape, except this time the top of the ladder didn't just fail to earn its tax, it actively lost.
Against Fable 5
My first instinct was that Fable 5 had walked away with it, since Fable's low-effort run was cheaper and faster than Opus 5 at high. That comparison was junk. I was putting one model's low effort against another's high.
Matched properly, at low effort on both, they're near parity: Opus 5 costs 1.15x and takes 1.22x the wall time, and scores slightly better on deterministic checks with zero code errors against Fable's two. Worth noting Fable's list price is double Opus 5's, so Opus is still burning more tokens per job. Just not enough to change the answer.
The same model cost 35% less on a different provider
Here's the one I didn't expect. I reran the winning config, low effort, against the first-party Anthropic API instead of OpenRouter. Same model, same effort, same 23 scenarios, same day.
| Opus 5, effort low | Anthropic direct | OpenRouter |
|---|
| Total cost | $9.05 | $13.99 |
| Deterministic checks | 87/106 | 89/106 |
| Code errors | 1 | 0 |
| Avg duration | 108s | 116s |
Quality is a wash. Cost isn't: 35% cheaper, and cheaper in 22 of the 23 scenarios, which is too consistent to be noise. Both runs made exactly 33 LLM calls, so it isn't doing less work either.
My first guess was that our own cost accounting was wrong, that OpenRouter just wasn't reporting cached tokens in the field we read. So I probed both providers with a repeated prefix, and I was wrong: OpenRouter reports cache reads correctly. What the probe did show is that its top-level caching writes the prefix at the 1.25x rate the first time it sees one. Across 23 mostly independent scenarios, those writes don't get read back often enough to pay for themselves. That last step is inference rather than something I measured per-request, but the direction is consistent across almost every scenario.
Worth knowing if you route through a gateway for convenience. The gateway isn't neutral on cost, even when it's the same model underneath and the token prices on paper are identical.
What I'm actually shipping
Gemini 3.6 Flash stays the default for now. It's still cheaper than Opus 5 at low effort with a comparable pass rate, and cost per generation is the number that decides this for a product where people iterate. The provider finding narrows that gap though: against the $9.05 first-party run rather than the $13.99 gateway one, Gemini's lead shrinks from roughly 2.7x to 1.7x.
The thing tugging the other way is the error column. Gemini threw 8 code errors across the suite, Opus 5 at low effort threw zero or one depending on the provider. A code error isn't just a retry on our side, it's a user watching a spinner while the agent repairs itself. I'm not switching on one 23 scenario run, but that gap is what I'll be watching. If I do switch, it'll be the first-party route.
Every run, including the ugly ones, is logged on the evals page. If you want to see what the models actually produce, the 3D viewer will open any result you export.
