Back to Home

Gestalt's 1,931 Elo: What the Design Benchmark Hides

A model called Gestalt sits atop a design leaderboard nobody had heard of a week ago, and its maker has not put a name on it. Contra Labs published the first Frontier Design Benchmark standings on November 18, and the stealth entry leads with 1,931 Elo, 218 points clear of Claude Fable 5.2. Those ratings are public. The rollout budget behind them is not.

The benchmark is a real piece of work rather than a launch-day chart. Contra Labs built it with BenchFlow from 33 briefs: 27 rewritten from real design jobs posted on Contra, plus six projects from startup partners who had already hired a Contra designer. The 27 came out of an analysis of roughly 4,600 website projects and cover audience, page flows, brand assets and practical constraints such as mobile-first layouts and fast loads.

Generation runs through BenchFlow, which spins up an agent in a sandbox per task and records the output, cost and full trajectory. OpenRouter serves the models, and Contra says each model ran in its own lab's agent harness wherever possible. Human golden outputs came from a tighter loop: three moodboard directions, reviewer feedback, then a build in whatever tools the designer preferred. The comparison is therefore not model against model. It is model plus harness plus an unpublished budget.

Judging is blind and pairwise. Thirty-five professional designers cast 4,311 votes over five rounds, each seeing two designs for the same brief with names hidden and sides shuffled. Contra turns those votes into ratings with a Bradley-Terry model expressed as Elo, then splits the panel into random halves and ranks the field twice. Across 200 splits the two rankings agree at Spearman 0.95, a healthy number for a panel this small.

The top is one model, the next seven are a tie

Read the standings as intervals rather than ranks and the shape changes. Gestalt's 90% win rate comes with a 95% confidence interval of 86 to 92 on 420 votes. Behind it, the field compresses into a band that a few unlucky votes could reshuffle:

  • Gestalt (stealth): 1,931 Elo, 90% wins, CI 86 to 92, 420 votes.
  • Human designers recruited from Contra: 1,739 Elo, 74% wins, CI 71 to 78, 571 votes.
  • Claude Fable 5.2: 1,713 Elo, 73% wins, CI 69 to 76, 683 votes.
  • Claude Opus 5.5: 1,691 Elo, 73% wins, CI 69 to 76, 527 votes.
  • GPT-6.1 Sol: 1,668 Elo, 70% wins, CI 66 to 74, 549 votes.

Ranks two through eight span 101 Elo points and their intervals overlap almost end to end. Calling Claude Fable 5.2 clearly better than Kimi K3.1 Pro at 1,663 is not supportable from these numbers; calling Gestalt clearly better than everything else is. That is the only separation the board establishes.

The Elo arithmetic is checkable, which most leaderboards do not allow. A 218 point gap implies a 77.8% expected win rate, and Contra reports 78% against Fable 5.2, so the headline fits the rating exactly. The 192 point gap over human work implies about 75%, while the disclosed human matchup reads 20 of 28 votes, or 71.4%. At 28 ballots that miss is unremarkable.

Confidence width has a second reading. Gestalt holds fewer votes than Fable 5.2, 420 against 683, yet a tighter interval. A 90% win rate has smaller binomial variance than a 73% rate, so extremes narrow their own error bars. The narrow interval is a consequence of dominance, not proof of it.

Three metrics, three different winners

Contra grades the same outputs a second and third way. Its Design Critique Rubric covers 35 principles across typography, layout, consistency, copy, usability and accessibility. On rounded scores, Claude Fable 5.2 and Claude Opus 5.5 each take 84 out of 100. Gestalt takes 83. The model that wins human preference loses the checklist by a point.

That split is the most useful thing on the board. Pairwise preference rewards what a judge notices: a striking hero section, confident typography, a layout that reads as expensive. A rubric rewards the absence of faults, and the benchmark's own human example is a button with too little contrast plus a wordmark half hidden by the section above it. Crowd-pleasing and fault-free are different skills.

The rubric is not decorative. Contra reports that rubric scores track the pairwise ranking at r = 0.94 and Spearman 0.95 across the field, and a separate Slop Detector scores outputs for AI tells. Agreement at field level with disagreement at the top is what you would expect if the top entrant is optimized for impression.

The column that is missing: cost per win

BenchFlow already records cost and trajectory for every run, yet the leaderboard publishes no cost, no retry budget and no rule for choosing which attempt got submitted. A studio can send its first draft or its fifth favorite, and both are legitimate submissions. Only one of those habits survives contact with a real invoice.

There is a governance question too. Contra co-founder Gajus Kuizinas says Contra did not build Gestalt, and the research team says it works with the model's developers and gives them feedback. Blind voting protects the comparison from name recognition, not from unequal conditions behind each ballot, and full dataset access is granted on request rather than published.

None of this makes the 1,931 fake. It makes it incomplete. The test that would settle it is boring to run: one unseen brief, one fixed budget, one selection rule for every entrant, then a single client revision. Report the cost of reaching a design the client accepts. Until a leaderboard carries that number next to the Elo, the rank tells you which output people liked, not what it cost to get there.

Comments

No comments yet. Be the first to share your thoughts!