First full run for Gemini 3

High initial, code fix medium

The rest is the same as a648be9

Main reason of this run is to see how it compares with default service tier (a173ee9).

Running GPT 5.1 Medium to see how it performs on the base set

No visual checks, example in context

Medium reasoning, low 5.1 for code fixes

With 16k thinking budget (4x more than the base)

No visual checks, example in context

2.5-flash for visual eval, tuned confidence, predicted human score

Running it for the second time with the same configuration to see if it has the same results.

GPT-5 with reasoning effort set to medium

Via google vertex

promptCacheRetention set to 24h to see how it improves the duration

Extended, with low reasoning for code fixes

Includes a 600 line example model in the prompt.

GPT 120B OSS for code fixing

Running again with medium to see if there is any difference

Examples, no visual checks

2.5-flash for visual eval, tuned confidence, predicted human score

Same as `GPT-5 base run` but with `reasoning_effort` set to `low`.

No reasoning for edit or initial

High reasoning for initial prompts. NONE for code fixing prompts.

Auto thinking budget

second run with updated system prompt (there was a really common failure)

Smaller dataset just a preliminary check

Running with priority service tier to see if duration reduces

First run was with medium, this is with high reasoning

Rerunning with new system prompt that forces it to generate more stuff ask less questions

gpt-5.1-codex-mini for fixes, caching set to 24h

Medium reasoning effort for fixing the visual errors

Running extended set to see how it performs.

This includes the base run to benchmark against. 7 prompts, 3 times each.

Running a smaller benchmark on SOTA open source model