Experiments / Experiment 05

Experiment 05 · A Prompt test, start to finish · Tested 4 November 2026

Can AI write marketing emails without making things up?

The best-value run passed 89% of answers at $0.57 per 1,000. The top run passed 100% at $2.67, about 4.7 times the price.

Their likely ranges overlap, so this test alone does not show a clear lead. What a likely range is ↓

Opens a new Prompt test with this experiment's prompt, checks and models. Paste yours over it. New here? Your email starts a free trial. No credit card needed.

Best value
89% at $0.57 per 1,000 answers
DeepSeek V4.1 Flash, medium thinking. 32 of 36 answers passed, likely range 67–97%.
Top score
100% at $2.67 per 1,000 answers
GPT-6.1 Sol, medium thinking. 36 of 36 answers passed, likely range 82–100%.
One model, same setting, new prompt
0% → 75%
0 of 36 → 27 of 36 passed. Only the prompt changed.

Ask a model for a marketing email and it wants to sell. We tested seven models on briefs for invented shops. With a plain prompt, most answers added an offer, a date or a claim nobody gave it.

A test case, not a customer. We built this on invented data and ran it on 4 November 2026. Models and prices change, and we don't rerun these often, so check the date.

What we do with prompts

This page is one run of it.

  1. StartYour prompt

    The one you already use. Here, a plain first draft.

  2. 1 · ScoreChecks and test cases

    7 checks and 24 test cases, written for the task.

  3. 1 · ScoreEvery answer scored

    756 answers, each checked against every rule.

  4. 2 · CompareOther models

    7 models, with cost and speed, to find the same result cheaper or faster.

  5. 3 · ImproveRewrite, score again

    0% → 75% passed on one model, same setting (0 → 27 of 36).

Step 1Score

We wrote the checks and test cases, and scored every answer

  1. 24test cases, 18 kept unseen for scoring
  2. 7checks on every answer
  3. 756answers scored, on 4 November 2026
  4. 61%the best any model passed on the first prompt (22 of 36)

The checks

Must-pass checks one miss fails the answer

  • Uses only offers, dates and products stated in the briefmust pass
  • Makes no claim about results, health or scarcity the brief doesn't supportmust pass
  • The body is at most 100 wordsmust pass
  • The subject is at most 50 charactersmust pass
  • Has one clear call to actionmust pass
  • Valid JSON with subject, bodymust pass

Quality checks scored, but never fail an answer

  • Keeps to the tone the brief asks fornice to have

An answer passes only if it meets every must-pass check. Quality checks are scored and shown check by check further down, so better wording never hides a broken rule.

The first prompt 144 characters
You write marketing emails for small online shops. Write an email from this brief. Return JSON with the keys subject and body.

Brief: {{brief}}
Step 2Compare

The same test on other models: who won

Best valueDeepSeek V4.1 Flash, medium thinking

32 of 36 at $0.57 per 1,000 answers, in 5.0 s.

Best qualityGPT-6.1 Sol, low thinking

35 of 36 at $1.96. Medium got 36 at $2.67.

Cheapest worth tryingClaude Haiku 5.5, medium thinking

29 of 36 at $0.49.

Passes against cost, after the rewriteOff or low thinking Medium thinking
09182736 $0.01$0.10$1$10 Cost per 1,000 answers (log scale) Passed, of 36 Better ↖ GPT-6.1 Sol, medium thinking: 36 of 36, $2.67, 6.0 sGPT-6.1 Sol, low thinking: 35 of 36, $1.96, 4.0 sSol, low: best qualityClaude Opus 5.5, medium thinking: 34 of 36, $14.13, 7.3 sOpus, mediumDeepSeek V4.1 Flash, medium thinking: 32 of 36, $0.57, 5.0 sDeepSeek, medium: best valueClaude Haiku 5.5, medium thinking: 29 of 36, $0.49, 5.9 sGemini 3.8 Flash, medium thinking: 29 of 36, $5.41, 10.4 sMercury 3, medium thinking: 27 of 36, $0.80, 6.5 sClaude Opus 5.5, low thinking: 27 of 36, $9.30, 5.3 sNex N2.5 Mini, medium thinking: 24 of 36, $0.09, 3.8 sDeepSeek V4.1 Flash, off thinking: 23 of 36, $0.10, 1.2 sGemini 3.8 Flash, low thinking: 21 of 36, $1.18, 2.5 sNex N2.5 Mini, off thinking: 17 of 36, $0.01, 0.9 sClaude Haiku 5.5, off thinking: 17 of 36, $0.14, 2.0 sMercury 3, off thinking: 9 of 36, $0.07, 0.9 s

Hover over a point for the model and setting. Every run is in the table below.

All 14 runs
ModelThinkingFirst promptAfter one rewriteLikely rangeCost per 1,000 answersMedian timeThinking tokens
GPT-6.1 SolMediumnot run36 of 3682–100%$2.676.0 s117
GPT-6.1 SolLow2235 of 3678–99%$1.964.0 s45
Claude Opus 5.5Mediumnot run34 of 3674–99%$14.137.3 s433
DeepSeek V4.1 FlashMediumnot run32 of 3667–97%$0.575.0 s725
Claude Haiku 5.5Mediumnot run29 of 3658–93%$0.495.9 s713
Gemini 3.8 FlashMediumnot run29 of 3658–93%$5.4110.4 s1284
Mercury 3Mediumnot run27 of 3652–89%$0.806.5 s2088
Claude Opus 5.5Low027 of 3652–89%$9.305.3 s184
Nex N2.5 MiniMediumnot run24 of 3644–84%$0.093.8 s806
DeepSeek V4.1 FlashOff123 of 3641–82%$0.101.2 s0
Gemini 3.8 FlashLow021 of 3636–78%$1.182.5 s149
Nex N2.5 MiniOff217 of 3627–69%$0.010.9 s0
Claude Haiku 5.5Off017 of 3627–69%$0.142.0 s0
Mercury 3Off09 of 3611–48%$0.070.9 s0

Likely range. Each run answered 18 test cases twice. With so few cases a score is a rough figure. The range is where the true pass rate probably sits, 95 times in 100. 10 of the 14 runs have a range that overlaps the top run's, so this test does not clearly separate them. The rest are behind by more than chance. Test your own prompt on more cases before you choose.

Sorted by passes, then cost. First prompt was run at the lowest setting only. Thinking tokens are per answer, as each provider reports them. Cost is the model's price for the answer, per 1,000 answers, on the test date. A code fence around JSON counts as valid, but anything that doesn't parse is a fail.

Does more thinking help?

Thinking mattered a lot. DeepSeek went from 23 of 36 with thinking off to 32 on medium, Haiku from 17 to 29 and Mercury from 9 to 27. It wasn't cheap: Mercury used about 2,000 thinking tokens an answer, at 11 times the cost. Sol barely needed it, with 35 on low and 36 on medium.

Step 3Improve

We rewrote the prompt and scored it again

One model, same setting, only the prompt changed

First prompt0%
Rewrite75%

0 of 36 → 27 of 36 passed

The biggest gain of the 7 models, marked “headline” in the list, on 18 cases the rewrite never saw, run twice. Same thinking setting both times, same cases.

The rewrite raised 7 of 7 models. Each at its lowest thinking setting.

First prompt Rewrite

Claude Opus 5.5 headline0% → 75%
DeepSeek V4.1 Flash3% → 64%
Gemini 3.8 Flash0% → 58%
Claude Haiku 5.50% → 47%
Nex N2.5 Mini6% → 47%
GPT-6.1 Sol61% → 97%
Mercury 30% → 25%
First prompt 144 characters
You write marketing emails for small online shops. Write an email from this brief. Return JSON with the keys subject and body.

Brief: {{brief}}
After one rewrite 2073 characters
You write marketing emails for small online shops. Write an email from this brief. Return only valid JSON with exactly the keys "subject" and "body", both strings. Escape quotation marks, backslashes and line breaks correctly; do not use Markdown fences.

Keep the subject at most 50 characters and the entire body at most 100 words, including any greeting, sign-off and call to action. Shorter is fine, especially for a sparse brief.

Use only facts stated in the brief. Do not invent offers, discounts, codes, dates, deadlines, products, materials, features, stock, delivery terms, reasons for delays or absence, customer behaviour, or business history. Do not add unsupported benefits, results, health, safety, durability, growth or scarcity claims—even if they seem natural for the product. For example, reflective does not justify a visibility or safety claim, and wool does not justify claims of purity, warmth or durability. Preserve stated prices, dates and restrictions accurately. Leave missing details out rather than filling them in. A brief such as "Sale." does not supply discount amounts, scope, timing or urgency; "We're back." does not imply new stock, offers or a reason for the absence.

Include exactly one clear request for the reader to take one action in the body. Do not repeat it as a button or add another action in the sign-off. Choose an action suited to the brief: a shopping action for a promotion, or a simple invitation to reply for a thank-you or service update. If details are sparse, use a neutral invitation such as "Reply to say hello." Do not invent links, contact details or service promises.

Follow the requested tone and output language throughout, including greetings and sign-offs. Keep restrained messages restrained and cosy messages natural, not cheesy. For delays, apologise directly, preserve any uncertainty about dispatch, and avoid forced cheerfulness or unsupported assurances.

Before returning the JSON, check both length limits, factual support and that the body contains exactly one call to action.

Brief: {{brief}}

What changed in the prompt

  1. Say the brief is the only source

    Inventing details was the most common failure. Tell the model that an offer, date or product not in the brief doesn't exist.

  2. Ban the usual sales claims

    Models reached for lines about calm rooms and better mornings. Name the kinds of claim to avoid.

  3. Give hard limits

    Nearly half the first answers ran past 100 words. State the subject and body limits.

  4. Then check it still sells

    Our rewrite made some emails very plain. Add a check for appeal so the prompt doesn't win by saying nothing.

What it fixed, check by check

The share of answers that passed each check, across all the models with thinking at its lowest setting.

First prompt After one rewrite

Must-pass checks

Uses only offers, dates and products stated in the brief 25%73%
Makes no claim about results, health or scarcity the brief doesn't support 42%88%
The body is at most 100 words 54%98%
The subject is at most 50 characters 71%98%
Has one clear call to action 79%85%
Valid JSON with subject, body 82%94%

Quality checks

Keeps to the tone the brief asks for nice to have 90%89%

Real answers, before and after

How we tested it

  • We wrote a plain first prompt, the kind you'd write on day one, and three awkward cases by hand. A model drafted the rest, for 24 cases in all.
  • The prompt ran on 6 of those cases. GPT-6.1 Sol rewrote it once from what failed, which is what Spring Prompt's assistant does.
  • Both prompts then ran on the 18 cases the rewrite never saw, twice, so scores are out of 36. An answer passes only if it meets every must check.
  • We set thinking on purpose. Lowest means off where a model allows it, and low where it doesn't. We also ran the rewrite at medium.
  • GPT-6 Luna judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand.

What this can't tell you

  • The cases are invented, so a real product will behave differently. Your prompt, data and customers matter more than this table.
  • 18 cases is small. Gaps of two or three cases can be chance, so read the top few as level.
  • A model drafted most cases and a model judged them.
  • Costs are model prices for the answers alone, and they change often. Times come from one day's runs.
  • The checks reward sticking to the brief. They don't measure whether anyone would click, and some passing emails were dull.

How we know the judge is right

A model judged each answer with three votes. Word limits and JSON were checked by code, and we read a sample of answers by hand. The three votes are there to catch a judge that changes its mind. Code checks what can be counted. Reading by hand is how we look for a judge that is wrong the same way every time.

A judge can still be wrong, and this experiment has no agreement figure of its own. In your own Prompt test you can open any answer and agree with the judge or give your own verdict, and the test shows how often the judge agreed with you. How often our judges agree with people →

Free until 23 November 2026 · No credit card needed

Want a hand setting it up?

Tell us what your prompt does. We'll set the test up with you, inside your own free trial account, so your prompt stays in your account. You don't send it to us.

  1. Score. Checks and test cases written for your task, and every answer scored.
  2. Compare. The same test on other models, to see if you get the same result cheaper or faster.
  3. Improve. A rewrite from what failed, scored again on the same cases.

We reply by email. See our privacy policy.