The first test found no winner. The second found that the models that check their own work are the ones that land. For the third I picked a problem where a wrong answer is impossible to hide — deterministic chaos, with the referee built into the file — and watched what the models did when correctness was no longer the thing that separated them.
The story so far
Test 01 was a Fourier visualizer in one HTML file, with a live error readout as a built-in referee. Four models, identical prompt, first pass only. All four shipped working math within a rounding error of each other. The only thing that varied was the bill, by ~200×. The narrow, honest conclusion: for a well-specified task with a checkable answer, model choice stops being the variable that matters.
Test 02 was a playable Lunar Lander with two referees — exact free-fall physics, and an open-ended autopilot that had to fly the model's own game. The physics tied again. The autopilot didn't: 20/20, 18/20, 5/20, 0/20. And the thing that predicted the split wasn't the model's name or price — it was whether the model tested its own work before saying done.
So for test 03 I wanted a task that was fully checkable — no open-ended half this time — but where the checkable answer is genuinely hard to fake. If the "model doesn't matter" thesis is real, this is where it should be at its strongest. And I wanted to see what's left to separate four models when even correctness converges.
The answer turned out to be: chaos.
double-pendulum long-exposure trace
The test: a referee you can't argue with
Same rules as always. One prompt, pasted into four CLIs at once, four folders, first pass only, terminals recorded.
The prompt: a double pendulum in one self-contained HTML file. Fixed constants (two 1 kg bobs, two 1 m rods, g = 9.81), a fixed starting position (both arms at 120°, released from rest), the full nonlinear equations of motion, RK4 at a 1/2000 s timestep. Draw it, trace the lower bob, and — the point of the whole thing — grade its own physics live, on screen, two ways:
- Energy drift. A frictionless double pendulum conserves total mechanical energy exactly. So the app computes its own energy every frame and reports how far it has drifted from the starting value. Any drift is pure integrator error. There is an analytic right answer, and it is zero. This is the referee that cannot be argued with.
- The Lyapunov exponent. The app runs a hidden twin pendulum, identical except its first angle is nudged by one part in a billion (10⁻⁹). It measures how fast the two drift apart, fits the exponential, and reports the growth rate λ. A positive λ is the mathematical signature of deterministic chaos — proof that the wild motion on screen is real sensitivity to initial conditions, not a numerical glitch.
One button runs both checks headless over 60 simulated seconds and prints the verdict.
Why a double pendulum? Because it is the cleanest example of a system that is
completely deterministic and completely unpredictable at the same time
. The equations have no randomness. Run them twice from the same start and you get the same motion every time. Nudge the start by a billionth of a radian and within fifteen seconds the two are doing entirely different things. It's the perfect stress test for the thesis: the physics is exactly checkable, but the behavior is impossible to eyeball.
double pendulums
To grade the graders, I built my own
There's a trap in a self-grading test: what if all four apps agree on a number that's simply wrong? A shared bug would look like consensus.
So before the race I wrote my own double-pendulum simulation from scratch — same constants, same RK4, same 60 seconds — as an independent answer key. It says: starting energy 14.715 J, maximum drift over 60 seconds 6.6×10⁻⁸ % (that's a rounding whisker away from perfect), largest Lyapunov exponent ≈1.4 per second, and the twin trajectories cross one full radian apart at ≈14.5 seconds.
That's the ground truth every app below is measured against. Not "the models agreed with each other" — "the models agreed with an independent simulation I control."
The race
Four terminals, four different tools: Sonnet 5 in Claude Code, Grok 4.5 and Composer 2.5 through the Grok CLI, DeepSeek V4 Pro in opencode.
Two of them treated it like a sprint. Composer 2.5 finished first, in 1 minute 43 seconds — wrote the file, printed a tidy "verified headless results" table, done. Grok 4.5 followed at with the most thorough write-up of the four: the cleanest textbook energy equation, and two separation metrics where everyone else reported one.
The other two treated it like an exam they had to hand in. Sonnet 5 wrote the most compact file of the field and then opened a real browser — fifteen times — to actually run its own referee. Along the way it noticed the automation tab was being throttled by Chrome in the background, reasoned out loud that its headless check didn't depend on the animation, ran it anyway, and only then called it done, at . DeepSeek V4 Pro did the same kind of thing through a different tool — drove a headless Chrome, clicked its own Run Referee button, took a screenshot of the panel to confirm — and finished at , having spent all of eight cents.
four terminals, four ai agents
Here's the tell, and it flips test 02 on its head. Composer and Grok reported correct referee numbers without ever opening the file. As far as the recording shows, they asserted their "validated" results from reasoning alone. Sonnet and DeepSeek reported correct numbers because they ran the thing and read the panel
Maybe I need to force playwright or chrome e2e next time.
The results, from each app's own referee
Referee #1 — energy drift, the exact half. Ground truth: 6.6×10⁻⁸ %.
Model
Max energy drift (self-reported)
Verdict
Sonnet 5
6.57×10⁻⁸ %
PASS
Grok 4.5
≈6.6×10⁻⁸ %
PASS
Composer 2.5
6.6×10⁻⁸ %
PASS
DeepSeek V4 Pro
4.94×10⁻⁸ % *
PASS
Converged. A third time. All four land on the same eight-decimal answer as my independent simulation. Nobody faked the integrator — you can't fake it, that's the whole design, and this time nobody even tried. (* DeepSeek reads a touch lower only because its headless check samples energy every 100th step instead of every step — the least precise verifier of the four, though still comfortably passing.)
Referee #2 — the Lyapunov exponent, the chaos half. Ground truth: ≈1.4 per second, positive.
Model
Fitted λ
1-radian divergence
Sonnet 5
1.634 /s
14.85 s
DeepSeek V4 Pro
1.535 /s (R² = 0.964)
14.50 s
Composer 2.5
1.47 /s
14.5 s
Grok 4.5
1.45 /s
14.85 s
All positive. All within the window each model chose to fit. The chaos is real, measured four independent ways, and it agrees with the answer key.
energy-drift
So what actually separated them?
Read the two tables and the honest answer is: on the thing the test measures, nothing did. All four are correct. This is the strongest the "model doesn't matter" thesis has looked in three tests — the task was fully checkable, genuinely hard, and every model nailed it.
But test 02 taught me to look at the column that isn't the score. Last time it was the autopilot. This time it's the clock, and the browser.
- The two fast models (Composer 1:43, Grok 3:58) shipped correct files they never ran.
- The two slow models (Sonnet 9:07, DeepSeek 9:15) shipped correct files because they ran them.
Here's the twist from test 02. Last time, the models that didn't verify failed — Grok's autopilot crashed 15 of 20 landings. This time, the models that didn't verify passed anyway. Composer and Grok were right without checking.
Which sharpens the lesson instead of repeating it: verification isn't what makes the output correct. It's what lets you know it's correct before you ship it. On a fully-checkable task, a good enough model can skip the check and still be right. It just can't know it's right. Composer and Grok got there in a quarter of the time — and shipped blind. Sonnet and DeepSeek paid five minutes and eight-cents-to-two-dollars for something the sprinters didn't buy: certainty.
Whether that trade is worth it depends entirely on the cost of being wrong. For a throwaway visualization, ship blind and save the minutes. For anything where a silent integrator bug reaches production, the model that opens the browser is the one you want — and DeepSeek proved that habit costs four cents, not two dollars.
Blind mode: where this test gets its teeth
There's a reason I picked chaos. Put the four pendulums side by side and let them run. Within fifteen seconds every trace is a different unpredictable scribble — and you cannot tell which model wrote which. The house style, the variable names, the little UI flourishes all vanish the moment the physics takes over. Four correct implementations of a chaotic system are, to the eye, indistinguishable.
That's not a bug in the test. That is the finding, rendered as a picture: when the task is fully specified and correctly solved, the model's fingerprint disappears. Try the blind mode and see if you can beat a coin flip. I couldn't.
four double-pendulum traces
What the bill bought
- DeepSeek V4 Pro: $0.0432, itemized to the token. It was also one of the two that drove a real browser to grade itself. Cheapest and most diligent — the value pick, again, by two orders of magnitude.
- Sonnet 5: ~$2.02, token-metered (it ran on a plan, so this is my token count × published pricing). Most of it went to the fifteen-call verification loop that produced the field's most careful sign-off.
- Grok 4.5 and Composer 2.5: flat subscription through the Grok CLI, no per-token price — so no honest dollar figure, and no bar on my chart.
The caveats, before you quote me
One run per model. The four CLIs were launched a minute or two apart, so the stopwatches in the video are each model's true wall-clock from its own logs, not a synchronized gun — the durations are honest, the "start line" is staggered, and I'd rather tell you that than fake a clean start. DeepSeek's headless referee undersamples, which is why its drift number reads lowest; I'm calling that a minor imprecision, not an error, because its live panel samples every frame and my independent run confirms the real value. And "verification buys confidence, not correctness" is a claim about this task, where the answer was fully checkable — on an open-ended task (see test 02) verification bought correctness too.
Everything is published and runs live. You don't have to trust a number in this article — press the button yourself.
https://x.com/0xBakeer/status/2077442955934101680
Try it
All four untouched files run in your browser. Hit Play to watch the chaos, hit Run Referee to reproduce every number above:
- The test: https://khaledbakeer.github.io/One-Prompt/tests/double-pendulum/index.html
- The arena (all four, side by side): https://khaledbakeer.github.io/One-Prompt/tests/double-pendulum/arena.html
- Blind mode — guess who built which: https://khaledbakeer.github.io/One-Prompt/tests/double-pendulum/blind.html
- Test 02, the Lunar Lander round: https://khaledbakeer.github.io/One-Prompt/tests/lunar-lander/index.html
- Test 01, the Fourier round: https://khaledbakeer.github.io/One-Prompt/tests/fourier-epicycles/index.html
- Raw data, prompt included: https://khaledbakeer.github.io/One-Prompt/data/double-pendulum.json
The energy table is the control group — it's supposed to tie, and it does. The clock is the experiment. And the blind mode is the punchline: four correct answers to a chaotic problem look exactly alike.
— Khaled
