GrandpaCAD: AI-Powered 3D Modeling for Everyone | GrandpaCAD

This dashboard benchmarks each model and CAD-engine combo on a fixed set of prompts, scoring prompt adherence, aesthetics, cost, speed, and code errors. The short version: medium reasoning is the value sweet spot, Opus 4.7 is our reliability benchmark, and cheap models usually cost more once you count error-repair loops and latency.

Sort by:

23 × 1 = 23 runs

Gemini 3.7 Flash — cheap suite (no judge)

model gemini-3.7-flash, thinking medium

gemini-3.7-flashThinking mediumopenscad0.2

21 × 1 = 21 runs

Opus 4.7 with one visual fix

Attempting with one visual fix

openscad0.2active

23 × 1 = 23 runs

openscad engine (1 attempt)

model gemini-3.5-flash, thinking low

gemini-3.5-flashThinking lowopenscad0.2

23 × 1 = 23 runs

Claude Opus 5 (Anthropic direct, effort low)

model claude-opus-5-anthropic, thinking adaptive, effort low

claude-opus-5-anthropicThinking lowopenscad0.2

23 × 1 = 23 runs

Gemini 3.7 Flash — HIGH reasoning — cheap suite (no judge)

model gemini-3.7-flash, thinking high

gemini-3.7-flashThinking highopenscad0.2

23 × 1 = 23 runs

gemini-3.6-flash openscad thinking medium (1 attempt, no judge)

model gemini-3.6-flash, thinking medium

gemini-3.6-flashThinking mediumopenscad0.2

23 × 1 = 23 runs

OpenSCAD errors baseline — run1 (1 attempt, no judge)

model gemini-3.1-pro, thinking medium

gemini-3.1-proThinking mediumopenscad0.2

23 × 1 = 23 runs

gemini-3.5-flash openscad thinking medium (no judge)

model gemini-3.5-flash, thinking medium

gemini-3.5-flashThinking mediumopenscad0.2

23 × 1 = 23 runs

Claude Opus 5 (OpenRouter, reasoning low)

model claude-opus-5, reasoning low

claude-opus-5Thinking lowopenscad0.2

23 × 1 = 23 runs

kimi-k3 benchmark (1 attempt, no judge)

model kimi-k3, reasoning low

kimi-k3Thinking lowopenscad0.2

23 × 1 = 23 runs

opus-4.6 benchmark (1 attempt, no judge)

model claude-opus-4.6, thinking adaptive, effort low

claude-opus-4.6Thinking lowopenscad0.2

23 × 1 = 23 runs

opus-4.8 benchmark (1 attempt, no judge)

model claude-opus-4.8, thinking adaptive, effort low

claude-opus-4.8Thinking lowopenscad0.2

23 × 1 = 23 runs

sonnet-5 benchmark (1 attempt, no judge)

model claude-sonnet-5, thinking adaptive, effort low

claude-sonnet-5Thinking lowopenscad0.2

23 × 1 = 23 runs

Claude Opus 5 (OpenRouter, reasoning high)

model claude-opus-5, reasoning high

claude-opus-5Thinking highopenscad0.2

23 × 1 = 23 runs

model grok-4.5, reasoning low

grok-4.5Thinking lowopenscad0.2

23 × 1 = 23 runs

GPT-5.6 Sol (high reasoning)

model gpt-5.6-sol, reasoning high

gpt-5.6-solThinking highopenscad0.2

What we've learned

2025-10Medium reasoning beats High: about 2x the cost and time for little quality gain.
2025-11Nearly-free open models mostly failed or errored out. Sticker price is not real cost.
2025-12GPT-5.2 High was a cost trap: $0.65 per generation and 22 code errors.
2026-02Switched the target engine to OpenSCAD and began deliberate error-mining runs.
2026-04Opus 4.7 set the bar: fast, about $0.10 per generation, few code errors.
2026-06The cheapest models (like minimax-m3) hit 90+ code errors and multi-hour runs.
2026-07gemini-3.1-pro is the current OpenSCAD baseline.
2026-07Opus 5 at low reasoning beat itself at high on every axis: 28% cheaper, 2.4x faster, 5 code errors down to 0, same adherence.
2026-07Same Opus 5, same effort, 35% cheaper on the first-party Anthropic API than through OpenRouter. The gateway is not cost-neutral.

Other Stats

Average R-squared

8.95%

What is the average R-squared of all the runs? This tells us linear regression alignment human vs AI. 100% means AI perfectly predicts a human vote. 0% means AI doesn't predict it at all.

Combined R-squared

17.14%

When we set the model according to all pairs from all runs what is the R-squared?

Eval Runs

97

Total number of evaluation runs.

Total Cost of All Evals

$665.77

The sum of costs for all evaluation runs.

Average Cost per Test Run

$6.86

The average cost of a single evaluation run.

Total Duration of All Evals

6553m 17s

The sum of durations for all evaluation runs.

Average Duration per Test Run

67m 34s

The average duration of a single evaluation run.

All Generations

2988

The sum of all generations for all evaluation runs.