Experiments / Experiment 02

Experiment 02 · A Prompt test, start to finish · Tested 4 November 2026

Which model should reply to your ecommerce customers?

The best-value run passed 89% of answers at $0.39 per 1,000. The top run passed 100% at $1.50, about 3.8 times the price.

Their likely ranges overlap, so this test alone does not show a clear lead. What a likely range is ↓

Opens a new Prompt test with this experiment's prompt, checks and models. Paste yours over it. New here? Your email starts a free trial. No credit card needed.

Best value
89% at $0.39 per 1,000 answers
DeepSeek V4.1 Flash, medium thinking. 32 of 36 answers passed, likely range 67–97%.
Top score
100% at $1.50 per 1,000 answers
GPT-6.1 Sol, low thinking. 36 of 36 answers passed, likely range 82–100%.
One model, same setting, new prompt
0% → 94%
0 of 36 → 34 of 36 passed. Only the prompt changed.

A support reply can sound perfect and still promise a refund nobody agreed to. We tested seven models on invented orders. With a basic prompt, the most expensive model passed none of them.

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we do with prompts

This page is one run of it.

  1. StartYour prompt

    The one you already use. Here, a plain first draft.

  2. 1 · ScoreChecks and test cases

    7 checks and 24 test cases, written for the task.

  3. 1 · ScoreEvery answer scored

    756 answers, each checked against every rule.

  4. 2 · CompareOther models

    7 models, with cost and speed, to find the same result cheaper or faster.

  5. 3 · ImproveRewrite, score again

    0% → 94% passed on one model, same setting (0 → 34 of 36).

Step 1Score

We wrote the checks and test cases, and scored every answer

  1. 24test cases, 18 kept unseen for scoring
  2. 7checks on every answer
  3. 756answers scored, on 4 November 2026
  4. 64%the best any model passed on the first prompt (23 of 36)

The checks

Must-pass checks one miss fails the answer

  • At most 110 wordsmust pass
  • Hands chargebacks, legal threats and safety issues to a personmust pass
  • Promises no refund, replacement or date the order and policy don't supportmust pass
  • Applies the returns policy correctly to this ordermust pass
  • Names one clear next stepmust pass
  • Every order fact it states matches the order detailsmust pass

Quality checks scored, but never fail an answer

  • Acknowledges how the customer feels in one short sentencenice to have

An answer passes only if it meets every must-pass check. Quality checks are scored and shown check by check further down, so better wording never hides a broken rule.

The first prompt 261 characters
You are a customer support agent for an online store. Reply to the customer using the order details. Our policy: returns within 30 days of delivery, unworn, and the customer pays return postage unless the item is faulty.

Order: {{order}}

Customer: {{message}}
Step 2Compare

The same test on other models: who won

Best valueDeepSeek V4.1 Flash, medium thinking

32 of 36 at $0.39 per 1,000 answers, at a median of 6.0 s.

Best qualityGPT-6.1 Sol, low thinking

36 of 36 at $1.50. The choice if replies go out with no review.

Fastest worth usingMercury 3, medium thinking

29 of 36 in 2.1 s at $0.25, if an agent checks each reply.

Passes against cost, after the rewriteOff or low thinking Medium thinking
09182736 $0.01$0.10$1$10 Cost per 1,000 answers (log scale) Passed, of 36 Better ↖ GPT-6.1 Sol, low thinking: 36 of 36, $1.50, 4.5 sSol, low: best qualityGPT-6.1 Sol, medium thinking: 36 of 36, $1.96, 6.3 sClaude Opus 5.5, low thinking: 34 of 36, $8.01, 5.4 sOpus, lowDeepSeek V4.1 Flash, medium thinking: 32 of 36, $0.39, 6.0 sDeepSeek, medium: best valueClaude Haiku 5.5, medium thinking: 32 of 36, $0.44, 6.3 sGemini 3.8 Flash, low thinking: 32 of 36, $0.76, 3.2 sGemini 3.8 Flash, medium thinking: 32 of 36, $5.31, 11.0 sClaude Opus 5.5, medium thinking: 31 of 36, $11.52, 7.0 sMercury 3, medium thinking: 29 of 36, $0.25, 2.1 sMercury, medium: fastestDeepSeek V4.1 Flash, off thinking: 25 of 36, $0.06, 1.6 sNex N2.5 Mini, medium thinking: 23 of 36, $0.06, 5.0 sMercury 3, off thinking: 17 of 36, $0.06, 0.8 sClaude Haiku 5.5, off thinking: 15 of 36, $0.13, 2.5 sNex N2.5 Mini, off thinking: 13 of 36, $0.01, 1.1 s

Hover over a point for the model and setting. Every run is in the table below.

All 14 runs
ModelThinkingFirst promptAfter one rewriteLikely rangeCost per 1,000 answersMedian timeThinking tokens
GPT-6.1 SolLow2336 of 3682–100%$1.504.5 s5
GPT-6.1 SolMediumnot run36 of 3682–100%$1.966.3 s51
Claude Opus 5.5Low034 of 3674–99%$8.015.4 s144
DeepSeek V4.1 FlashMediumnot run32 of 3667–97%$0.396.0 s584
Claude Haiku 5.5Mediumnot run32 of 3667–97%$0.446.3 s651
Gemini 3.8 FlashLow132 of 3667–97%$0.763.2 s42
Gemini 3.8 FlashMediumnot run32 of 3667–97%$5.3111.0 s1267
Claude Opus 5.5Mediumnot run31 of 3664–96%$11.527.0 s327
Mercury 3Mediumnot run29 of 3658–93%$0.252.1 s562
DeepSeek V4.1 FlashOff125 of 3646–86%$0.061.6 s0
Nex N2.5 MiniMediumnot run23 of 3641–82%$0.065.0 s547
Mercury 3Off017 of 3627–69%$0.060.8 s0
Claude Haiku 5.5Off015 of 3622–64%$0.132.5 s0
Nex N2.5 MiniOff713 of 3618–59%$0.011.1 s0

Likely range. Each run answered 18 test cases twice. With so few cases a score is a rough figure. The range is where the true pass rate probably sits, 95 times in 100. 11 of the 14 runs have a range that overlaps the top run's, so this test does not clearly separate them. The rest are behind by more than chance. Test your own prompt on more cases before you choose.

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

Does more thinking help?

This task showed the largest effect. Haiku went from 15 of 36 with thinking off to 32 on medium, at 3.4 times the cost. Mercury went from 17 to 29, and DeepSeek from 25 to 32 at 6.5 times the cost. At the top it made no difference or went the wrong way: Sol scored 36 on both, Opus fell from 34 to 31, and Gemini stayed at 32 for 7 times the cost.

Step 3Improve

We rewrote the prompt and scored it again

One model, same setting, only the prompt changed

First prompt0%
Rewrite94%

0 of 36 → 34 of 36 passed

The biggest gain of the 7 models, marked “headline” in the list, on 18 cases the rewrite never saw, run twice. Same thinking setting both times, same cases.

The rewrite raised 7 of 7 models. Each at its lowest thinking setting.

First prompt Rewrite

Claude Opus 5.5 headline0% → 94%
Gemini 3.8 Flash3% → 89%
DeepSeek V4.1 Flash3% → 69%
Mercury 30% → 47%
Claude Haiku 5.50% → 42%
GPT-6.1 Sol64% → 100%
Nex N2.5 Mini19% → 36%
First prompt 261 characters
You are a customer support agent for an online store. Reply to the customer using the order details. Our policy: returns within 30 days of delivery, unworn, and the customer pays return postage unless the item is faulty.

Order: {{order}}

Customer: {{message}}
After one rewrite 1158 characters
You are a customer support agent for an online store. Reply to the customer using the order details in at most 110 words. Acknowledge the customer's feelings in one short sentence, not just a thank-you or an apology for the problem.

Our policy: returns within 30 days of delivery, unworn, and the customer pays return postage unless the item is faulty. Day 30 is included. For a reported fault, explain that the customer does not pay return postage. Do not invent rules about trying items on, tags or packaging; if unworn status is unclear, do not confirm eligibility.

State only supported order facts. This policy does not specify refunds, replacements, payment methods, processing times or return procedures: do not promise or invent them, including labels, addresses or actions already taken. Do not guarantee a requested date.

Give one clear next step, not a menu of options. Where return instructions or eligibility need confirmation, ask the customer to contact a support person for that confirmation rather than inventing details. Refer chargebacks, legal threats and safety issues to a human support agent.

Order: {{order}}

Customer: {{message}}

What changed in the prompt

  1. Spell out what the policy doesn't cover

    Models invented refund routes and timescales. Tell the model what isn't specified, and not to fill it in.

  2. Pin down the edge cases

    State whether day 30 counts, and what happens with faulty items. Applying the policy was still the most common failure after the rewrite.

  3. Ask for one next step

    A list of options fails the check. Tell the model to give one clear action.

  4. Then test for helpfulness

    Our rewrite made models cautious. Many passing replies simply sent customers to a person, so add a check that the reply solves what it can.

What it fixed, check by check

The share of answers that passed each check, across all the models with thinking at its lowest setting.

First prompt After one rewrite

Must-pass checks

At most 110 words 35%94%
Hands chargebacks, legal threats and safety issues to a person 35%95%
Promises no refund, replacement or date the order and policy don't support 51%97%
Applies the returns policy correctly to this order 64%78%
Names one clear next step 87%94%
Every order fact it states matches the order details 90%93%

Quality checks

Acknowledges how the customer feels in one short sentence nice to have 41%89%

Real answers, before and after

How we tested it

  • We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
  • The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
  • Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
  • We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
  • GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

  • The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
  • 18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
  • A model drafted most cases and a model judged them.
  • Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
  • The checks reward caution. A pass means the reply broke no rule, not that it was the most helpful reply possible.

How we know the judge is right

A model judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand. The three votes are there to catch a judge that changes its mind. Code checks what can be counted. Reading by hand is how we look for a judge that is wrong the same way every time.

A judge can still be wrong, and this experiment has no agreement figure of its own. In your own Prompt test you can open any answer and agree with the judge or give your own verdict, and the test shows how often the judge agreed with you. How often our judges agree with people →

Free until 23 November 2026 · No credit card needed

Want a hand setting it up?

Tell us what your prompt does. We'll set the test up with you, inside your own free trial account, so your prompt stays in your account. You don't send it to us.

  1. Score. Checks and test cases written for your task, and every answer scored.
  2. Compare. The same test on other models, to see if you get the same result cheaper or faster.
  3. Improve. A rewrite from what failed, scored again on the same cases.

We reply by email. See our privacy policy.