Running first time with OpenSCAD Eval

First full run for Gemini 3

Reverting back to Gemini 3 with the everything the same as Gemini 3.1 run so we can compare the costs and stuff.

High initial, code fix medium

Not much except for gathering errors

We run a first "big" error analysis and improved context based on it.

This is run before we will run the analysis of errors for improvement flow

We have improved the prompt, let's see the improvement

The rest is the same as a648be9

Main reason of this run is to see how it compares with default service tier (a173ee9).

Single generation for each model - no variance considered

Running GPT 5.1 Medium to see how it performs on the base set

No visual checks, example in context

Medium reasoning, low 5.1 for code fixes

With 16k thinking budget (4x more than the base)

No visual checks, example in context

2.5-flash for visual eval, tuned confidence, predicted human score

Running it for the second time with the same configuration to see if it has the same results.

GPT-5 with reasoning effort set to medium

Via google vertex

promptCacheRetention set to 24h to see how it improves the duration

Extended, with low reasoning for code fixes

Includes a 600 line example model in the prompt.

GPT 120B OSS for code fixing

Running again with medium to see if there is any difference

Examples, no visual checks

2.5-flash for visual eval, tuned confidence, predicted human score

Trying to see if Gemini 3.1 is much better than Gemini 3.0.

Same as `GPT-5 base run` but with `reasoning_effort` set to `low`.

No reasoning for edit or initial

High reasoning for initial prompts. NONE for code fixing prompts.

Auto thinking budget

second run with updated system prompt (there was a really common failure)

Smaller dataset just a preliminary check

Running with priority service tier to see if duration reduces

First run was with medium, this is with high reasoning

Rerunning with new system prompt that forces it to generate more stuff ask less questions

gpt-5.1-codex-mini for fixes, caching set to 24h

Medium reasoning effort for fixing the visual errors

Running extended set to see how it performs.

This includes the base run to benchmark against. 7 prompts, 3 times each.

Running a smaller benchmark on SOTA open source model