Experiments / Experiment 06

Experiment 06 · A Prompt test, start to finish · Tested 4 November 2026

Which model should answer your phones?

The best-value run passed 100% of answers at $0.48 per 1,000. The next model to match it cost $1.65, about 3.4 times the price.

Opens a new Prompt test with this experiment's prompt, checks and models. Paste yours over it. New here? Your email starts a free trial. No credit card needed.

Best value
100% at $0.48 per 1,000 answers
DeepSeek V4.1 Flash, medium thinking. 36 of 36 answers passed, likely range 82–100%.
Next to match it
100% at $1.65 per 1,000 answers
GPT-6.1 Sol, medium thinking. 36 of 36 answers passed, likely range 82–100%.
One model, same setting, new prompt
0% → 100%
0 of 36 → 36 of 36 passed. Only the prompt changed.

On a phone call, a long answer is a bad answer. We tested seven models as a clinic receptionist. The first prompt nearly always ran too long, and one rewrite fixed most of that.

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we do with prompts

This page is one run of it.

  1. StartYour prompt

    The one you already use. Here, a plain first draft.

  2. 1 · ScoreChecks and test cases

    7 checks and 24 test cases, written for the task.

  3. 1 · ScoreEvery answer scored

    756 answers, each checked against every rule.

  4. 2 · CompareOther models

    7 models, with cost and speed, to find the same result cheaper or faster.

  5. 3 · ImproveRewrite, score again

    0% → 100% passed on one model, same setting (0 → 36 of 36).

Step 1Score

We wrote the checks and test cases, and scored every answer

  1. 24test cases, 18 kept unseen for scoring
  2. 7checks on every answer
  3. 756answers scored, on 4 November 2026
  4. 42%the best any model passed on the first prompt (15 of 36)

The checks

Must-pass checks one miss fails the answer

  • At most 40 wordsmust pass
  • Asks at most one questionmust pass
  • Offers only the open slots and states only the listed pricemust pass
  • Never says “*” or “#”must pass
  • Gives no medical advice, and passes urgent symptoms to a personmust pass
  • Sounds natural spoken aloud, with no listsmust pass

Quality checks scored, but never fail an answer

  • Confirms the detail back before moving onnice to have

An answer passes only if it meets every must-pass check. Quality checks are scored and shown check by check further down, so better wording never hides a broken rule.

The first prompt 203 characters
You are the phone receptionist for a dental clinic. Help the caller. Open slots: Tue 9:30, Tue 14:00, Thu 11:15. New patient check-up costs £60. Say your reply as you would on a call.

Caller: {{caller}}
Step 2Compare

The same test on other models: who won

Best valueDeepSeek V4.1 Flash, medium thinking

36 of 36 at $0.48 per 1,000 answers, in 1.7 s.

FastestDeepSeek V4.1 Flash, thinking off

31 of 36 in 0.7 s at $0.06. Mercury off matched that score in 0.8 s.

Also perfectClaude Opus 5.5, low thinking

36 of 36, but at $6.66 and 4.3 s.

Passes against cost, after the rewriteOff or low thinking Medium thinking
09182736 $0.01$0.10$1$10 Cost per 1,000 answers (log scale) Passed, of 36 Better ↖ DeepSeek V4.1 Flash, medium thinking: 36 of 36, $0.48, 1.7 sDeepSeek, medium: best valueGPT-6.1 Sol, medium thinking: 36 of 36, $1.65, 4.3 sGemini 3.8 Flash, medium thinking: 36 of 36, $3.58, 7.3 sClaude Opus 5.5, low thinking: 36 of 36, $6.66, 4.3 sOpus, lowClaude Opus 5.5, medium thinking: 36 of 36, $9.68, 6.0 sClaude Haiku 5.5, medium thinking: 35 of 36, $0.30, 4.4 sGemini 3.8 Flash, low thinking: 35 of 36, $0.96, 2.7 sMercury 3, medium thinking: 33 of 36, $0.18, 1.5 sGPT-6.1 Sol, low thinking: 33 of 36, $1.18, 3.4 sMercury 3, off thinking: 31 of 36, $0.04, 0.8 sDeepSeek V4.1 Flash, off thinking: 31 of 36, $0.06, 0.7 sDeepSeek, off: fastestNex N2.5 Mini, medium thinking: 22 of 36, $0.05, 2.1 sNex N2.5 Mini, off thinking: 21 of 36, $0.01, 0.7 sClaude Haiku 5.5, off thinking: 18 of 36, $0.09, 1.9 s

Hover over a point for the model and setting. Every run is in the table below.

All 14 runs
ModelThinkingFirst promptAfter one rewriteLikely rangeCost per 1,000 answersMedian timeThinking tokens
DeepSeek V4.1 FlashMediumnot run36 of 3682–100%$0.481.7 s352
GPT-6.1 SolMediumnot run36 of 3682–100%$1.654.3 s59
Gemini 3.8 FlashMediumnot run36 of 3682–100%$3.587.3 s846
Claude Opus 5.5Low036 of 3682–100%$6.664.3 s169
Claude Opus 5.5Mediumnot run36 of 3682–100%$9.686.0 s320
Claude Haiku 5.5Mediumnot run35 of 3678–99%$0.304.4 s444
Gemini 3.8 FlashLow535 of 3678–99%$0.962.7 s147
Mercury 3Mediumnot run33 of 3671–98%$0.181.5 s408
GPT-6.1 SolLow1533 of 3671–98%$1.183.4 s11
Mercury 3Off231 of 3664–96%$0.040.8 s0
DeepSeek V4.1 FlashOff331 of 3664–96%$0.060.7 s0
Nex N2.5 MiniMediumnot run22 of 3639–80%$0.052.1 s418
Nex N2.5 MiniOff821 of 3636–78%$0.010.7 s0
Claude Haiku 5.5Off118 of 3629–71%$0.091.9 s0

Likely range. Each run answered 18 test cases twice. With so few cases a score is a rough figure. The range is where the true pass rate probably sits, 95 times in 100. 11 of the 14 runs have a range that overlaps the top run's, so this test does not clearly separate them. The rest are behind by more than chance. Test your own prompt on more cases before you choose.

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

Does more thinking help?

For the smaller models thinking changed the result: Haiku went from 18 of 36 with thinking off to 35 on medium, and DeepSeek from 31 to 36 for about a second more. Opus was already perfect on low. Gemini on medium added 1 pass for nearly 4 times the cost, and 7.3 s is too slow for a call.

Step 3Improve

We rewrote the prompt and scored it again

One model, same setting, only the prompt changed

First prompt0%
Rewrite100%

0 of 36 → 36 of 36 passed

The biggest gain of the 7 models, marked “headline” in the list, on 18 cases the rewrite never saw, run twice. Same thinking setting both times, same cases.

The rewrite raised 7 of 7 models. Each at its lowest thinking setting.

First prompt Rewrite

Claude Opus 5.5 headline0% → 100%
Gemini 3.8 Flash14% → 97%
Mercury 36% → 86%
DeepSeek V4.1 Flash8% → 86%
GPT-6.1 Sol42% → 92%
Claude Haiku 5.53% → 50%
Nex N2.5 Mini22% → 58%
First prompt 203 characters
You are the phone receptionist for a dental clinic. Help the caller. Open slots: Tue 9:30, Tue 14:00, Thu 11:15. New patient check-up costs £60. Say your reply as you would on a call.

Caller: {{caller}}
After one rewrite 1402 characters
You are the phone receptionist for a dental clinic. Help the caller. Open slots: Tue 9:30, Tue 14:00, Thu 11:15. New patient check-up costs £60. Say your reply as you would on a call.

Keep every reply to at most 40 words. Use natural spoken sentences, with no lists, asterisks or hash signs. Ask at most one question, requesting only one thing.

First briefly confirm the caller's relevant detail before moving on. If they only greet you, greet them back without inventing details.

For routine appointment or price enquiries, offer just one listed slot, matching their preference where possible. State £60 when discussing a new patient check-up. Never invent fees, services, opening hours, waiting lists or alternative availability. Keep each slot attached to its listed weekday; do not call it today or tomorrow. If their requested time is unavailable, acknowledge that, offer one listed slot without reading out alternatives, and ask whether another day could work. Do not claim an appointment is booked or changed before confirmation.

For urgent symptoms or injuries, prioritise passing the caller to a person: briefly repeat the key details and say you will pass them to a member of the dental team now. Give no medical advice, self-care instructions or treatment recommendations. Do not offer routine slots, promise immediate treatment or add a price to this urgent handoff.

Caller: {{caller}}

What changed in the prompt

  1. Write for the ear

    Set a hard word limit and ban lists and markdown. Opus used asterisks in a spoken reply.

  2. One question at a time

    Callers can't answer two at once. The first prompt often asked for several details together.

  3. Only offer what's listed

    Models worked out prices and totals nobody gave them. Tell the agent to say it can't confirm instead.

  4. Time matters as much as score

    Median time here is for text only. A real call adds speech recognition and voice on top.

What it fixed, check by check

The share of answers that passed each check, across all the models with thinking at its lowest setting.

First prompt After one rewrite

Must-pass checks

At most 40 words 17%89%
Asks at most one question 73%98%
Offers only the open slots and states only the listed price 83%91%
Never says “*” or “#” 88%99%
Gives no medical advice, and passes urgent symptoms to a person 88%97%
Sounds natural spoken aloud, with no lists 93%97%

Quality checks

Confirms the detail back before moving on nice to have 72%84%

Real answers, before and after

How we tested it

  • We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
  • The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
  • Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
  • We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
  • GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

  • The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
  • 18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
  • A model drafted most cases and a model judged them.
  • Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
  • This tests text replies. We didn't test speech recognition, voices or interruptions, which matter just as much on a real call.

How we know the judge is right

A model judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand. The three votes are there to catch a judge that changes its mind. Code checks what can be counted. Reading by hand is how we look for a judge that is wrong the same way every time.

A judge can still be wrong, and this experiment has no agreement figure of its own. In your own Prompt test you can open any answer and agree with the judge or give your own verdict, and the test shows how often the judge agreed with you. How often our judges agree with people →

Free until 23 November 2026 · No credit card needed

Want a hand setting it up?

Tell us what your prompt does. We'll set the test up with you, inside your own free trial account, so your prompt stays in your account. You don't send it to us.

  1. Score. Checks and test cases written for your task, and every answer scored.
  2. Compare. The same test on other models, to see if you get the same result cheaper or faster.
  3. Improve. A rewrite from what failed, scored again on the same cases.

We reply by email. See our privacy policy.