Benchmarks / tau2-bench
Reported by tau2-bench
tau2-bench
Share of retail orders, returns and exchanges tasks completed within policy, averaged over trials.
- Results dated
- 24 Feb 2026 to 26 May 2026
- Models
- 8
- Unit
- % of tasks
- Licence
- MIT License
Full results
| # | Model | tau2 v1.0.1, retail: task success (pass^1) % of tasks, higher is better |
|---|---|---|
| 1 | Qwen3.5 397B A17B (thinking reasoning, tau2)Alibaba |
84.4%
|
| 2 | GPT-5.2 (high reasoning, tau2)OpenAI |
81.6%
|
| 3 | Claude Opus 4.5 (high reasoning, tau2)Anthropic |
79.6%
|
| 4 | gemini-3-flash (high reasoning, tau2)Google |
76.8%
|
| 5 | gemini-3-pro (high reasoning, tau2)Google |
75.9%
|
| 6 | GPT-5.2 (no reasoning, tau2)OpenAI |
75.0%
|
| 7 | GLM 5 (thinking reasoning, tau2)Z.ai |
73.7%
|
| 8 | Claude Sonnet 4.5 (thinking reasoning, tau2)Anthropic |
72.4%
|
Results as published by tau2-bench; we do not re-run them.
What it measures
Share of retail orders, returns and exchanges tasks completed within policy, averaged over trials.
What it does not measure
Not your policies or customers: the tasks and policies are tau2's own, and the customer is simulated by gpt-5.2.
Source
Failures
Failures count against a model: a product with no usable output is a failed listing. They are listed here so you can see why.
Not ranked
- Distyl ButtonAgent: custom agent scaffold, not the standard harness
- RAFT-30B-A3B: custom agent scaffold, not the standard harness
tau2-bench by Sierra Research and contributors, MIT License.