Experiments / Experiment 03

Experiment 03 · A Prompt test, start to finish · Tested 4 November 2026

Which model writes meeting notes you can trust?

The best-value run passed 92% of answers at $0.38 per 1,000. The top run passed 97% at $2.54, about 6.7 times the price.

Their likely ranges overlap, so this test alone does not show a clear lead. What a likely range is ↓

Opens a new Prompt test with this experiment's prompt, checks and models. Paste yours over it. New here? Your email starts a free trial. No credit card needed.

Best value
92% at $0.38 per 1,000 answers
Claude Haiku 5.5, medium thinking. 33 of 36 answers passed, likely range 71–98%. One cheaper run scored about the same, but it came from the model that judged the answers, so we do not lead with it. It is in the table below.
Top score
97% at $2.54 per 1,000 answers
GPT-6.1 Sol, medium thinking. 35 of 36 answers passed, likely range 78–99%.
One model, same setting, new prompt
31% → 81%
11 of 36 → 29 of 36 passed. Only the prompt changed.

The risk in meeting notes isn't a clumsy summary. It's a decision nobody made, or a task with an owner who never agreed. We tested eight models on invented meetings, where people change their minds.

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we do with prompts

This page is one run of it.

  1. StartYour prompt

    The one you already use. Here, a plain first draft.

  2. 1 · ScoreChecks and test cases

    6 checks and 24 test cases, written for the task.

  3. 1 · ScoreEvery answer scored

    864 answers, each checked against every rule.

  4. 2 · CompareOther models

    8 models, with cost and speed, to find the same result cheaper or faster.

  5. 3 · ImproveRewrite, score again

    31% → 81% passed on one model, same setting (11 → 29 of 36).

Who agreed to that? How we tested a meeting-notes prompt, in about a minute.
Step 1Score

We wrote the checks and test cases, and scored every answer

  1. 24test cases, 18 kept unseen for scoring
  2. 6checks on every answer
  3. 864answers scored, on 4 November 2026
  4. 69%the best any model passed on the first prompt (25 of 36)

The checks

Must-pass checks one miss fails the answer

  • Lists as decisions only things the group actually decided, not ideas raisedmust pass
  • Names an owner only if the transcript names one, and otherwise leaves it emptymust pass
  • Gives a due date only if the transcript states onemust pass
  • Valid JSON with summary, decisions, action_itemsmust pass
  • Misses no action item someone clearly agreed to domust pass
  • Reflects the final position when people changed their mindsmust pass

An answer passes only if it meets every must-pass check.

The first prompt 173 characters
Summarise this meeting transcript. Return JSON with the keys summary, decisions and action_items, where each action item has task, owner and due.

Transcript: {{transcript}}
Step 2Compare

The same test on other models: who won

Best valueGPT-6 Luna, medium thinking

32 of 36 at $0.18 per 1,000 answers, at a median of 4.5 s. Haiku on medium got 33 at $0.38.

Best qualityGPT-6.1 Sol, medium thinking

35 of 36 at $2.54. Opus on low also got 35, at $9.14.

Fastest worth usingMercury 3, medium thinking

33 of 36 in 2.9 s at $0.48.

Passes against cost, after the rewriteOff or low thinking Medium thinking
09182736 $0.01$0.10$1$10 Cost per 1,000 answers (log scale) Passed, of 36 Better ↖ GPT-6.1 Sol, medium thinking: 35 of 36, $2.54, 4.6 sSol, medium: best qualityClaude Opus 5.5, low thinking: 35 of 36, $9.14, 4.9 sOpus, lowDeepSeek V4.1 Flash, medium thinking: 34 of 36, $0.55, 6.2 sClaude Haiku 5.5, medium thinking: 33 of 36, $0.38, 5.0 sMercury 3, medium thinking: 33 of 36, $0.48, 2.9 sGPT-6 Luna, medium thinking: 32 of 36, $0.18, 4.5 sLuna, medium: best valueGPT-6.1 Sol, low thinking: 32 of 36, $2.24, 4.5 sGemini 3.8 Flash, medium thinking: 32 of 36, $4.01, 7.7 sClaude Opus 5.5, medium thinking: 32 of 36, $14.04, 6.6 sGPT-6 Luna, off thinking: 31 of 36, $0.10, 2.5 sGemini 3.8 Flash, low thinking: 31 of 36, $0.83, 3.0 sClaude Haiku 5.5, off thinking: 30 of 36, $0.18, 2.3 sMercury 3, off thinking: 29 of 36, $0.09, 1.0 sDeepSeek V4.1 Flash, off thinking: 29 of 36, $0.11, 1.3 sNex N2.5 Mini, medium thinking: 22 of 36, $0.09, 4.1 sNex N2.5 Mini, off thinking: 13 of 36, $0.02, 1.7 s

Hover over a point for the model and setting. Every run is in the table below.

All 16 runs
ModelThinkingFirst promptAfter one rewriteLikely rangeCost per 1,000 answersMedian timeThinking tokens
GPT-6.1 SolMediumnot run35 of 3678–99%$2.544.6 s28
Claude Opus 5.5Low2135 of 3678–99%$9.144.9 s59
DeepSeek V4.1 FlashMediumnot run34 of 3674–99%$0.556.2 s871
Claude Haiku 5.5Mediumnot run33 of 3671–98%$0.385.0 s368
Mercury 3Mediumnot run33 of 3671–98%$0.482.9 s1191
GPT-6 LunaMediumnot run32 of 3667–97%$0.184.5 s155
GPT-6.1 SolLow2532 of 3667–97%$2.244.5 s6
Gemini 3.8 FlashMediumnot run32 of 3667–97%$4.017.7 s862
Claude Opus 5.5Mediumnot run32 of 3667–97%$14.046.6 s294
GPT-6 LunaOff1931 of 3664–96%$0.102.5 s0
Gemini 3.8 FlashLow2131 of 3664–96%$0.833.0 s6
Claude Haiku 5.5Off1830 of 3661–94%$0.182.3 s0
Mercury 3Off1129 of 3658–93%$0.091.0 s0
DeepSeek V4.1 FlashOff2429 of 3658–93%$0.111.3 s0
Nex N2.5 MiniMediumnot run22 of 3639–80%$0.094.1 s796
Nex N2.5 MiniOff1513 of 3618–59%$0.021.7 s0

Likely range. Each run answered 18 test cases twice. With so few cases a score is a rough figure. The range is where the true pass rate probably sits, 95 times in 100. 15 of the 16 runs have a range that overlaps the top run's, so this test does not clearly separate them. The rest are behind by more than chance. Test your own prompt on more cases before you choose.

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

Does more thinking help?

Medium thinking lifted most models, by 3 to 5 cases for Haiku, Mercury, DeepSeek and Sol, and it cost Mercury and DeepSeek about 5 times as much. Nex rose to 22 and stayed the weakest. Opus went the other way, from 35 on low to 32 on medium, at a higher price. Haiku on medium came within 2 of Opus on low at about a twentieth of the cost.

Step 3Improve

We rewrote the prompt and scored it again

One model, same setting, only the prompt changed

First prompt31%
Rewrite81%

11 of 36 → 29 of 36 passed

The biggest gain of the 8 models, marked “headline” in the list, on 18 cases the rewrite never saw, run twice. Same thinking setting both times, same cases.

The rewrite raised 7 of 8 models, and Nex N2.5 Mini did worse. Each at its lowest thinking setting.

First prompt Rewrite

Mercury 3 headline31% → 81%
Claude Opus 5.558% → 97%
GPT-6 Luna53% → 86%
Claude Haiku 5.550% → 83%
Gemini 3.8 Flash58% → 86%
GPT-6.1 Sol69% → 89%
DeepSeek V4.1 Flash67% → 81%
Nex N2.5 Mini42% → 36%
First prompt 173 characters
Summarise this meeting transcript. Return JSON with the keys summary, decisions and action_items, where each action item has task, owner and due.

Transcript: {{transcript}}
After one rewrite 1439 characters
Summarise this meeting transcript. Return only valid JSON, without markdown or other text, with the keys summary, decisions and action_items. Use a string for summary and arrays for decisions and action_items, where each action item has task, owner and due. Use empty arrays when there are no decisions or action items, even if the transcript is incomplete or contains only small talk.

List as decisions only outcomes explicitly agreed in the transcript. Do not list suggestions, possibilities, conditional offers, unsupported statements of agreement, or the absence of an agreement or assignment as decisions. Put relevant context about unresolved matters in the summary instead.

Include every action item someone clearly agreed to do, but do not turn suggestions, unmet needs or unaccepted assignments into action items. Name an owner only if the transcript explicitly identifies one; otherwise use an empty string, not "Unassigned". Give a due date only if the transcript explicitly states it for that task; otherwise use an empty string. Preserve the stated deadline without adding inferred days or borrowing deadlines from related tasks. Prerequisites and conditions are not due dates.

Reflect the final position when people revise or withdraw earlier statements. Do not carry forward superseded tasks, conditions or commitments, or attach a condition to a task unless the transcript explicitly does so.

Transcript: {{transcript}}

What changed in the prompt

  1. Define a decision

    Say a decision is something explicitly agreed. Models listed ideas and clarifications as decisions.

  2. Say what to do with no owner or date

    Use an empty value, not a guess. Models filled these in with "Unassigned" or a likely date.

  3. Later statements win

    Say that the final position replaces earlier ones, and that a prerequisite isn't a due date.

  4. Ask for raw JSON

    Opus wrapped its JSON in Markdown fences and added a note after it, which breaks a parser.

What it fixed, check by check

The share of answers that passed each check, across all the models with thinking at its lowest setting.

First prompt After one rewrite

Lists as decisions only things the group actually decided, not ideas raised 75%89%
Names an owner only if the transcript names one, and otherwise leaves it empty 83%98%
Gives a due date only if the transcript states one 91%98%
Valid JSON with summary, decisions, action_items 96%98%
Misses no action item someone clearly agreed to do 97%93%
Reflects the final position when people changed their minds 100%99%

Real answers, before and after

How we tested it

  • We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
  • The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
  • Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
  • We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
  • GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

  • The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
  • 18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
  • A model drafted most cases and a model judged them.
  • Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
  • We added GPT-6 Luna on 6 November, on the same cases and prompts. Luna is also the judge, so we checked it with GPT-6.1 Sol judging fresh answers: on medium, Luna got 31 of 36 and Haiku 32.

How we know the judge is right

A model judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand. The three votes are there to catch a judge that changes its mind. Code checks what can be counted. Reading by hand is how we look for a judge that is wrong the same way every time.

A judge can still be wrong, and this experiment has no agreement figure of its own. In your own Prompt test you can open any answer and agree with the judge or give your own verdict, and the test shows how often the judge agreed with you. How often our judges agree with people →

Free until 23 November 2026 · No credit card needed

Want a hand setting it up?

Tell us what your prompt does. We'll set the test up with you, inside your own free trial account, so your prompt stays in your account. You don't send it to us.

  1. Score. Checks and test cases written for your task, and every answer scored.
  2. Compare. The same test on other models, to see if you get the same result cheaper or faster.
  3. Improve. A rewrite from what failed, scored again on the same cases.

We reply by email. See our privacy policy.