Benchmarks / ROASBench
Measured by Spring Prompt
ROASBench
Given twelve months of paid-marketing decisions for a simulated brand, which models grow the business and which overspend?
- Results dated
- 29 Sep 2026
- Models
- 17
- Unit
- months of 12
- Licence
- Spring Prompt original
| # | Model | Months over budget months of 12, lower is better | Overall score score out of 100 | Return on ad spend profit per £1 spent | Cost of a run US dollars |
|---|---|---|---|---|---|
| 1 | Claude Fable 5.1Anthropic |
0
|
51.5 | £0.64 | $2.49 |
| 1 | Claude Opus 5.5Anthropic |
0
|
51.6 | £0.69 | $1.06 |
| 1 | Claude Sonnet 5.5Anthropic |
0
|
52.0 | £0.70 | $0.42 |
| 1 | DeepSeek V4 Pro 0423DeepSeek |
0
|
28.2 | £0.26 | $0.20 |
| 1 | Gemini 3.1 Pro PreviewGoogle |
0
|
43.5 | £0.42 | $0.52 |
| 1 | Gemini 3.5 Flash LiteGoogle |
0
|
25.4 | £0.01 | $0.0390 |
| 1 | Gemini 3.8 FlashGoogle |
0
|
51.4 | £0.65 | $0.16 |
| 1 | Muse Spark 1.3Meta |
0
|
34.1 | £0.29 | $0.18 |
| 1 | Kimi K3Moonshot AI |
0
|
44.1 | £0.50 | $1.01 |
| 1 | GPT-6 AstraOpenAI |
0
|
56.6 | £0.67 | $1.23 |
| 1 | GPT-6 LunaOpenAI |
0
|
53.4 | £0.66 | $0.0142 |
| 1 | GPT-6 SolOpenAI |
0
|
54.1 | £0.65 | $0.24 |
| 1 | Grok 4.7xAI |
0
|
36.6 | £0.23 | $0.42 |
| 1 | GLM 5.3Z.ai |
0
|
24.8 | £0.01 | $0.13 |
| 15 | Qwen3.8 Max (0902)Alibaba |
2
|
25.4 | £0.09 | $0.73 |
| 15 | Claude Haiku 4.5Anthropic |
2
|
17.5 | −£0.21 | $0.0298 |
| 17 | Mistral Medium 3.5Mistral |
3
|
15.2 | −£0.17 | $0.15 |
Each model runs at its provider's default reasoning setting. Some providers think at length by default and others barely at all, so this is what you get without tuning.
What it measures
- Budget allocation across channels, month by month
- Reacting to last month's results
- Staying within budget
What it does not measure
- Real ad performance: the market is a deterministic simulation
- Creative quality of the ad copy
- Any brand other than one invented skincare company
Method
- A deterministic simulator scores every plan, so runs are reproducible
- Critical failures (overspend, broken plans) are reported per model
- Scores are not comparable with the 2026 v1 results