Lightning: strength against thinking time
Each configuration's median time per move at 10 seconds plus 1 a move. Right of the line, a model thinks longer than the increment it earns and the clock drains.
Benchmarks / BulletBench
Measured by Spring Prompt
BulletBench
When every second of thinking time comes off the clock, which models are fast enough to still make good decisions?
- Results dated
- 1 Oct 2026
- Models
- 23
- Unit
- US dollars
- Licence
- Spring Prompt original
Full results
| # | Model | BulletBench Bullet 60s: cost per game US dollars, lower is better | Lightning 10+1 ladder Elo | Bullet 60s ladder Elo | Blitz 3+2 ladder Elo | Move time milliseconds | Lost on time % of games | Cost per game US dollars |
|---|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Luna (no reasoning)OpenAI |
$0.0016
|
– | 249 | – | – | – | – |
| 2 | Seed-2.0-Mini (minimal reasoning)Bytedance |
$0.0030
|
571 | 453 | – | 0.3 s | 0.0% | $0.0019 |
| 3 | Mistral Small 4 (no reasoning)Mistral |
$0.0039
|
571 | 367 | – | 0.6 s | 0.0% | $0.0041 |
| 4 | GPT-5.4 Nano (no reasoning)OpenAI |
$0.0050
|
197 | 33 | – | 1.0 s | 75.0% | $0.0025 |
| 5 | Gemini 3.5 Flash LiteGoogle |
$0.0058
|
892 | 780 | 707 | 0.9 s | 0.0% | $0.0070 |
| 6 | Gemini 3.1 Flash Lite (minimal reasoning)Google |
$0.0068
|
780 | 912 | – | 0.8 s | 0.0% | $0.0058 |
| 7 | Inkling Small (no reasoning)Thinkingmachines |
$0.0075
|
571 | 529 | – | 0.6 s | 0.0% | $0.0077 |
| 8 | Gemini 3.6 Flash (minimal reasoning)Google |
$0.0113
|
33 | 628 | – | 1.5 s | 91.7% | $0.0050 |
| 9 | Muse Spark 1.3 (minimal reasoning)Meta |
$0.0142
|
– | 33 | – | – | – | – |
| 10 | Gemini 3.8 FlashGoogle |
$0.0154
|
– | 529 | 1,126 | – | – | – |
| 11 | GLM 5.3Z.ai |
$0.0156
|
– | 33 | – | – | – | – |
| 12 | Claude Haiku 4.5Anthropic |
$0.0163
|
33 | 445 | 491 | 1.3 s | 83.3% | $0.0095 |
| 13 | Gemini 3.8 Flash (low reasoning)Google |
$0.0165
|
197 | 1,068 | – | 1.8 s | 83.3% | $0.0041 |
| 14 | GPT-5.4 Mini (no reasoning)OpenAI |
$0.0205
|
571 | 197 | – | 0.9 s | 0.0% | $0.0136 |
| 15 | GLM 5.3 (low reasoning)Z.ai |
$0.0220
|
– | 33 | – | – | – | – |
| 16 | Gemini 3.5 Flash (minimal reasoning)Google |
$0.0289
|
727 | 810 | – | 1.2 s | 33.3% | $0.0213 |
| 17 | GPT-6 Sol (no reasoning)OpenAI |
$0.0297
|
– | 445 | – | – | – | – |
| 18 | Mistral Medium 3.5 (no reasoning)Mistral |
$0.0354
|
529 | 572 | – | 0.5 s | 0.0% | $0.0491 |
| 19 | Mistral Medium 3.5Mistral |
$0.0420
|
529 | 385 | 555 | 0.5 s | 0.0% | $0.0297 |
| 20 | Gemini 3.1 Pro Preview (low reasoning)Google |
$0.0446
|
– | 529 | – | – | – | – |
| 21 | Kimi K3 (low reasoning)Moonshot AI |
$0.0540
|
– | 33 | – | – | – | – |
| 22 | GPT-6 AstraOpenAI |
$0.0943
|
– | 197 | 839 | – | – | – |
| 23 | GPT-6 Astra (low reasoning)OpenAI |
$0.10
|
– | 323 | – | – | – | – |
Each model runs at its provider's default reasoning setting and, where the provider offers a faster one, again at its fastest setting, listed separately. Blitz 3+2 was run at default settings only.
Too slow for the clock
These configurations lost their first four games on time, so they were stopped there and are not rated for that clock. Being too slow is the result, not a fault.
Bullet 60s
- Claude Fable 5.1 4.2 s a move
- Claude Fable 5.1 (low reasoning) 4.5 s a move
- Claude Opus 5.5 3.1 s a move
- Claude Opus 5.5 (low reasoning) 3.3 s a move
- Claude Sonnet 5.5 2.9 s a move
- Claude Sonnet 5.5 (low reasoning) 2.6 s a move
- DeepSeek V4 Pro 0423 5.0 s a move
- DeepSeek V4.1 Flash (low reasoning) 2.4 s a move
- GLM 4.7 Flash 9.7 s a move
- GLM 5.3 Flash (low reasoning) 2.2 s a move
- GPT-6 Luna 5.4 s a move
- GPT-6 Sol 3.9 s a move
- Gemini 3.1 Pro Preview 5.0 s a move
- Grok 4.7 4.6 s a move
- Grok 4.7 (low reasoning) 5.2 s a move
- Kimi K3 3.2 s a move
- MiMo-V2.6-Flash 4.7 s a move
- Muse Spark 1.3 7.0 s a move
- Qwen3.8 Max (0902) 3.0 s a move
- Qwen3.8 Max (0902) (minimal reasoning) 4.7 s a move
- qwen/qwen3.8-flash 3.8 s a move
Lightning 10+1
- Claude Fable 5.1 4.4 s a move
- Claude Fable 5.1 (low reasoning) 4.1 s a move
- Claude Opus 5.5 3.0 s a move
- Claude Opus 5.5 (low reasoning) 2.9 s a move
- Claude Sonnet 5.5 2.1 s a move
- DeepSeek V4 Pro 0423 6.2 s a move
- DeepSeek V4.1 Flash (low reasoning) 0.8 s a move
- GLM 4.7 Flash 4.4 s a move
- GLM 5.3 1.8 s a move
- GLM 5.3 (low reasoning) 1.1 s a move
- GLM 5.3 Flash (low reasoning) 0.9 s a move
- GPT-6 Astra 2.1 s a move
- GPT-6 Astra (low reasoning) 2.3 s a move
- GPT-6 Luna 2.9 s a move
- GPT-6 Luna (no reasoning) 2.4 s a move
- GPT-6 Sol 1.7 s a move
- GPT-6 Sol (no reasoning) 1.7 s a move
- Gemini 3.1 Pro Preview 2.7 s a move
- Gemini 3.1 Pro Preview (low reasoning) 2.5 s a move
- Gemini 3.8 Flash 2.3 s a move
- Grok 4.7 2.1 s a move
- Grok 4.7 (low reasoning) 2.1 s a move
- Kimi K3 1.4 s a move
- Kimi K3 (low reasoning) 1.7 s a move
- MiMo-V2.6-Flash 3.2 s a move
- Muse Spark 1.3 4.6 s a move
- Muse Spark 1.3 (minimal reasoning) 2.2 s a move
- Qwen3.8 Max (0902) 3.8 s a move
- Qwen3.8 Max (0902) (minimal reasoning) 3.9 s a move
- qwen/qwen3.8-flash 3.0 s a move
Blitz 3+2
- DeepSeek V4 Pro 0423 7.3 s a move
- GPT-6 Sol 7.4 s a move
- Grok 4.7 12.6 s a move
- Kimi K3 4.0 s a move
- Muse Spark 1.3 19.4 s a move
- Qwen3.8 Max (0902) 7.1 s a move
Real games: the clock at Lightning 10+1
Each model's clock after every move of one game. A fast model earns back more than it spends; a slow one runs out within a few moves, whatever the position.
Gemini 3.5 Flash LiteWon by checkmate after 28 moves
- e40.6 s10.4 s left
- Nf30.9 s10.4 s left
- Bc40.6 s10.9 s left
- O-O0.9 s10.9 s left
- c30.6 s11.4 s left
- d40.9 s11.4 s left
- Nxe51.0 s11.4 s left
- Nxc60.6 s11.8 s left
- Qxg40.9 s11.9 s left
- Qxc8+0.9 s12.0 s left
- … 18 more moves
Claude Opus 5.5Lost on time after 3 moves
- e43.5 s7.5 s left
- Nf33.5 s5.0 s left
- d43.3 s2.6 s left
More from the results
Bullet: strength against thinking time
The same at 60 seconds for the whole game, no increment. A 40-move game leaves 1.5 seconds a move.
How games are lost
Share of games lost on time, and moves with no legal reply, by clock.
| Model | On time, Lightning | On time, Bullet | Invalid moves, Lightning | Invalid moves, Bullet |
|---|---|---|---|---|
| Gemini 3.5 Flash LiteGoogle | 0.0% | 0.0% | 0.0% | 0.0% |
| Gemini 3.1 Flash Lite (minimal reasoning)Google | 0.0% | 8.3% | 0.2% | 0.0% |
| Gemini 3.5 Flash (minimal reasoning)Google | 33.3% | 25.0% | 0.6% | 0.5% |
| Seed-2.0-Mini (minimal reasoning)Bytedance | 0.0% | 0.0% | 0.8% | 0.4% |
| Mistral Small 4 (no reasoning)Mistral | 0.0% | 25.0% | 1.0% | 0.6% |
| GPT-5.4 Mini (no reasoning)OpenAI | 0.0% | 50.0% | 0.0% | 0.0% |
| Inkling Small (no reasoning)Thinkingmachines | 0.0% | 0.0% | 5.3% | 7.2% |
| Mistral Medium 3.5 (no reasoning)Mistral | 0.0% | 0.0% | 0.0% | 0.4% |
| Mistral Medium 3.5Mistral | 0.0% | 8.3% | 0.3% | 0.8% |
| Gemini 3.8 Flash (low reasoning)Google | 83.3% | 50.0% | 0.0% | 0.0% |
| GPT-5.4 Nano (no reasoning)OpenAI | 75.0% | 66.7% | 4.0% | 3.1% |
| Claude Haiku 4.5Anthropic | 83.3% | 25.0% | 0.0% | 0.3% |
| Claude Sonnet 5.5 (low reasoning)Anthropic | 91.7% | – | 0.0% | – |
| Gemini 3.6 Flash (minimal reasoning)Google | 91.7% | 41.7% | 0.0% | 0.3% |
How BulletBench works
- 1
The position
The model gets the position, the moves so far, its remaining clock and the legal moves.
- 2
The reply
It answers with one move. The whole API round trip, including any hidden reasoning, comes off its clock.
- 3
The engine
Stockfish replies instantly at the model's current ladder level.
- 4
The ladder
Win and the next game is a level up; lose and it is a level down; draw and it stays. The results give a rating with a 95% interval.
Why chess on a clock
Many products need an answer in about a second: routing, triage, autocomplete, live agents. BulletBench asks which models can still think usefully at that speed. Chess gives a hard, objective score, and the clock makes slow thinking a real cost instead of a free extra.
The clocks
- Lightning 10+1
- Ten seconds, plus one second a move. A model can play for ever, but only at about a second an answer.
- Bullet 60s
- One minute for the whole game, no increment.
- Blitz 3+2
- Three minutes plus two seconds a move: room for reasoning models, for comparison.
Fair play
- Same prompt
- Every model gets the same system prompt and position format.
- Invalid replies
- A reply with no legal move gets two corrective retries, then a random legal move is played and counted.
- Settings
- Each model runs at its provider's default and, where the provider offers one, its fastest reasoning setting, shown separately.
Honest caveats
Twelve games per clock give wide intervals: models whose ranges overlap share a rank. Latency depends on the provider's servers on the day. The ladder's Elo anchors come from v1's Stockfish; this edition runs Stockfish 19, so ratings are not comparable with v1's.
What it measures
- Decision quality under real time pressure
- Response latency at the provider's default settings
- Which reasoning settings are fast enough to use
What it does not measure
- Any business skill: this is chess against an engine
- Chess strength without a clock
Method
- Games against a Stockfish ladder; every second of response time comes off the model's clock
- Each model at its default setting and, where offered, its fastest reasoning setting
- Ladder Elo with a 95% interval; it orders models, it is not a FIDE rating