Experiments / Experiment 09

Experiment 09 · A Prompt test, start to finish · Tested 4 November 2026

Which model turns notes into an honest slide outline?

The best-value run passed 72% of answers at $0.87 per 1,000. The top run passed 81% at $5.57, about 6.4 times the price.

Their likely ranges overlap, so this test alone does not show a clear lead. What a likely range is ↓

Opens a new Prompt test with this experiment's prompt, checks and models. Paste yours over it. New here? Your email starts a free trial. No credit card needed.

Best value
72% at $0.87 per 1,000 answers
DeepSeek V4.1 Flash, medium thinking. 26 of 36 answers passed, likely range 49–88%.
Top score
81% at $5.57 per 1,000 answers
GPT-6.1 Sol, medium thinking. 29 of 36 answers passed, likely range 58–93%.
One model, same setting, new prompt
3% → 67%
1 of 36 → 24 of 36 passed. Only the prompt changed.

This was the hardest of our tests. With a plain prompt, almost nothing passed on any model, because models fill slides with claims the notes never made. Thinking made the biggest difference we've seen.

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we do with prompts

This page is one run of it.

  1. StartYour prompt

    The one you already use. Here, a plain first draft.

  2. 1 · ScoreChecks and test cases

    6 checks and 24 test cases, written for the task.

  3. 1 · ScoreEvery answer scored

    756 answers, each checked against every rule.

  4. 2 · CompareOther models

    7 models, with cost and speed, to find the same result cheaper or faster.

  5. 3 · ImproveRewrite, score again

    3% → 67% passed on one model, same setting (1 → 24 of 36).

Step 1Score

We wrote the checks and test cases, and scored every answer

  1. 24test cases, 18 kept unseen for scoring
  2. 6checks on every answer
  3. 756answers scored, on 4 November 2026
  4. 3%the best any model passed on the first prompt (1 of 36)

The checks

Must-pass checks one miss fails the answer

  • Each slide title states the point, not just the topicmust pass
  • Uses only numbers and facts in the notesmust pass
  • Ends with the decision or ask in the briefmust pass
  • Has at most 6 slides, with at most 3 points eachmust pass
  • Keeps any caveat in the notes next to the figure it qualifiesmust pass
  • Valid JSON with title, slidesmust pass

An answer passes only if it meets every must-pass check.

The first prompt 187 characters
Turn these notes into a slide outline for the audience in the brief. Return JSON with the keys title and slides, where each slide has title and points.

Brief: {{brief}}

Notes: {{notes}}
Step 2Compare

The same test on other models: who won

Best valueDeepSeek V4.1 Flash, medium thinking

26 of 36 at $0.87 per 1,000 answers, but 8.9 s.

Best qualityGPT-6.1 Sol, medium thinking

29 of 36 at $5.57 and 11.9 s.

Fast and cheapDeepSeek V4.1 Flash, thinking off

22 of 36 at $0.15 in 2.2 s.

Passes against cost, after the rewriteOff or low thinking Medium thinking
09182736 $0.01$0.10$1$10 Cost per 1,000 answers (log scale) Passed, of 36 Better ↖ GPT-6.1 Sol, medium thinking: 29 of 36, $5.57, 11.9 sSol, medium: best qualityDeepSeek V4.1 Flash, medium thinking: 26 of 36, $0.87, 8.9 sDeepSeek, medium: best valueClaude Opus 5.5, medium thinking: 26 of 36, $26.25, 10.8 sOpus, mediumGPT-6.1 Sol, low thinking: 24 of 36, $3.88, 6.3 sDeepSeek V4.1 Flash, off thinking: 22 of 36, $0.15, 2.2 sGemini 3.8 Flash, medium thinking: 20 of 36, $10.09, 15.1 sClaude Opus 5.5, low thinking: 19 of 36, $15.38, 6.4 sClaude Haiku 5.5, medium thinking: 15 of 36, $0.96, 8.2 sGemini 3.8 Flash, low thinking: 15 of 36, $1.59, 3.6 sMercury 3, medium thinking: 12 of 36, $0.93, 11.1 sNex N2.5 Mini, medium thinking: 11 of 36, $0.15, 7.1 sMercury 3, off thinking: 4 of 36, $0.14, 1.4 sClaude Haiku 5.5, off thinking: 3 of 36, $0.29, 3.1 sNex N2.5 Mini, off thinking: 2 of 36, $0.03, 1.9 s

Hover over a point for the model and setting. Every run is in the table below.

All 14 runs
ModelThinkingFirst promptAfter one rewriteLikely rangeCost per 1,000 answersMedian timeThinking tokens
GPT-6.1 SolMediumnot run29 of 3658–93%$5.5711.9 s241
DeepSeek V4.1 FlashMediumnot run26 of 3649–88%$0.878.9 s1740
Claude Opus 5.5Mediumnot run26 of 3649–88%$26.2510.8 s696
GPT-6.1 SolLow124 of 3644–84%$3.886.3 s66
DeepSeek V4.1 FlashOff022 of 3639–80%$0.152.2 s0
Gemini 3.8 FlashMediumnot run20 of 3634–75%$10.0915.1 s2338
Claude Opus 5.5Low019 of 3631–73%$15.386.4 s197
Claude Haiku 5.5Mediumnot run15 of 3622–64%$0.968.2 s1331
Gemini 3.8 FlashLow015 of 3622–64%$1.593.6 s61
Mercury 3Mediumnot run12 of 3616–56%$0.9311.1 s3543
Nex N2.5 MiniMediumnot run11 of 3614–54%$0.157.1 s1768
Mercury 3Off04 of 363–33%$0.141.4 s0
Claude Haiku 5.5Off03 of 362–29%$0.293.1 s0
Nex N2.5 MiniOff02 of 361–26%$0.031.9 s0

Likely range. Each run answered 18 test cases twice. With so few cases a score is a rough figure. The range is where the true pass rate probably sits, 95 times in 100. 9 of the 14 runs have a range that overlaps the top run's, so this test does not clearly separate them. The rest are behind by more than chance. Test your own prompt on more cases before you choose.

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

Does more thinking help?

Medium thinking helped almost every model. Haiku went from 3 of 36 to 15, Nex from 2 to 11, and Opus from 19 to 26. It was also the most expensive setting we've tested, with Mercury using over 3,500 thinking tokens an answer. Some answers came back blank because the model used its whole budget thinking.

Step 3Improve

We rewrote the prompt and scored it again

One model, same setting, only the prompt changed

First prompt3%
Rewrite67%

1 of 36 → 24 of 36 passed

The biggest gain of the 7 models, marked “headline” in the list, on 18 cases the rewrite never saw, run twice. Same thinking setting both times, same cases.

The rewrite raised 7 of 7 models. Each at its lowest thinking setting.

First prompt Rewrite

GPT-6.1 Sol headline3% → 67%
DeepSeek V4.1 Flash0% → 61%
Claude Opus 5.50% → 53%
Gemini 3.8 Flash0% → 42%
Mercury 30% → 11%
Claude Haiku 5.50% → 8%
Nex N2.5 Mini0% → 6%
First prompt 187 characters
Turn these notes into a slide outline for the audience in the brief. Return JSON with the keys title and slides, where each slide has title and points.

Brief: {{brief}}

Notes: {{notes}}
After one rewrite 1890 characters
Turn these notes into a slide outline for the audience in the brief. Return only valid JSON with the keys title and slides, where each slide has title and points. Make title a string, slides an array, and points an array of strings.

Use at most 6 slides and at most 3 points per slide, including the final slide. Follow any smaller slide count requested in the brief.

Each slide title must state its takeaway as a specific claim or instruction, not name a topic. For example, use “Arrive 15 minutes before your shift and sign in,” not “Arrival and Sign-In,” when supported by the notes.

Use the brief to choose the audience, tone, structure and closing decision or ask. Use only the notes for factual content and practical guidance. Do not add plausible details, explanations, benefits, examples, advice, promises or contact arrangements. Preserve who does what and any conditions or exceptions; do not make instructions more specific than the notes. Keep each caveat or exception next to the figure or rule it qualifies, preferably in the same point.

If the notes are empty or information is missing, explicitly identify what is not provided rather than inventing content or using placeholders that imply confirmed arrangements. Keep any requested slide count within the limits above, using missing-information statements where necessary.

End with the decision or ask explicitly specified in the brief. Do not replace it with a summary, thanks, slogan or unrelated instruction. If the brief specifies no decision or ask, make the final point “No decision or ask is specified in the brief.” Do not invent one.

Before returning the JSON, check every slide title states a point, every factual claim and instruction is supported, all caveats stay beside what they qualify, the slide and point limits hold, and the final point follows the closing rule.

Brief: {{brief}}

Notes: {{notes}}

What changed in the prompt

  1. Make slide titles state the point

    Nearly every first answer used topic labels, such as "Next steps". Ask for a sentence that says what the slide shows.

  2. Forbid outside knowledge

    Models explained how heat pumps work, or added benefits, when the notes didn't say so. This stayed the top failure.

  3. Tie caveats to figures

    Keep each caveat next to the number it qualifies, and say so in the prompt.

  4. Set the size

    Give the number of slides and points, or models will run long.

What it fixed, check by check

The share of answers that passed each check, across all the models with thinking at its lowest setting.

First prompt After one rewrite

Each slide title states the point, not just the topic 6%100%
Uses only numbers and facts in the notes 8%44%
Ends with the decision or ask in the brief 38%75%
Has at most 6 slides, with at most 3 points each 54%98%
Keeps any caveat in the notes next to the figure it qualifies 84%98%
Valid JSON with title, slides 87%98%

Real answers, before and after

How we tested it

  • We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
  • The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
  • Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
  • We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
  • GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

  • The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
  • 18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
  • A model drafted most cases and a model judged them.
  • Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
  • No model passed more than 29 of 36, so have a person review every outline.
  • We tested outlines as text, not finished slides or designs.

How we know the judge is right

A model judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand. The three votes are there to catch a judge that changes its mind. Code checks what can be counted. Reading by hand is how we look for a judge that is wrong the same way every time.

A judge can still be wrong, and this experiment has no agreement figure of its own. In your own Prompt test you can open any answer and agree with the judge or give your own verdict, and the test shows how often the judge agreed with you. How often our judges agree with people →

Free until 23 November 2026 · No credit card needed

Want a hand setting it up?

Tell us what your prompt does. We'll set the test up with you, inside your own free trial account, so your prompt stays in your account. You don't send it to us.

  1. Score. Checks and test cases written for your task, and every answer scored.
  2. Compare. The same test on other models, to see if you get the same result cheaper or faster.
  3. Improve. A rewrite from what failed, scored again on the same cases.

We reply by email. See our privacy policy.