Docs Build with
The API: test prompts and read the results from your own code
The API does what the app and our MCP server do, as plain JSON over HTTPS: start a test of a prompt on several models, read what it found, improve the prompt, and ask which model each of your saved tests points to. It is for servers, scripts and scheduled jobs. If you work through a coding agent, the guide for agents covers the same actions as MCP tools.
The typical use is a job that reads your saved tests' recommendations and keeps the model each feature of your product uses up to date. There is an example below.
Get a key
- Sign in and open Connect your agent. A free trial account is enough to try it. No credit card needed.
- Make a key. It is shown once. Keep it in your server's secrets, never in a repo or a prompt.
- Send it with every call as a bearer token:
Authorization: Bearer YOUR_KEY.
The same keys work for the MCP server and for checks from a pull request. Removing a key on that page stops it at once. Calling the API needs the Pro plan. The free trial includes it while the trial lasts; the plans page says what each plan includes. A call from an account without it is refused with 402 and spends nothing.
The examples below read the key from the environment:
export SPRINGPROMPT_KEY="your key"
The endpoints
Every address starts with https://springprompt.com/api/v1. Calls that send data take a JSON object, and every answer is a JSON object. The answers shown here are examples of the shape: the model ids and the figures are made up.
| Call | What it does | Credit |
|---|---|---|
GET /api/v1/models | The models a test can use, with list prices per million tokens, and the credit left on the account. | Free |
POST /api/v1/tests | Starts a test of one prompt on several models and answers at once. | Spends credit, never more than max_spend_usd by estimate |
GET /api/v1/tests/{test_id} | A test's status and, when it is done, what it found. | Free |
GET /api/v1/tests | The account's recent Prompt tests, newest first. | Free |
GET /api/v1/tests/{test_id}/recommendation | Which model one saved test's answers point to, read from the answers the test already holds, with what has changed since it last ran. | Free: it runs nothing |
PATCH /api/v1/tests/{test_id} | Sets the lowest pass rate a saved test may recommend a model at, from 0 to 100 percent of its cases, or null to take it off. | Free: it runs nothing |
GET /api/v1/recommendations | The same for every saved test on the account: the model sheet as JSON. | Free: it runs nothing |
POST /api/v1/tests/{test_id}/improve | Rewrites a finished test's prompt from its failures and your ratings, adds it as a new version and tests it on the same cases, checks and models. | Spends credit: about one more run of the test |
GET /api/v1/judges | The account's trained judges: each learned from one person's ratings and says how often it agreed with them on answers it had not seen. | Free |
POST /api/v1/judges/{judge_id}/score | Scores up to 50 answers with a trained judge: how likely the person it learned from is to call each one good, from 0 to 1. | Spends a little credit |
GET /api/v1/models
The models a test can use, with list prices per million tokens, and the credit left on the account. Free.
curl -sS "https://springprompt.com/api/v1/models" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with credit_left_usd, the starting_mix, models with prices per million tokens, and a note on how many more there are.
{
"credit_left_usd": 18.4,
"starting_mix": [
"maker-a/model-large",
"maker-b/model-small"
],
"models": [
{
"id": "maker-a/model-large",
"name": "Model Large",
"input_per_million": 3.0,
"output_per_million": 15.0
},
{
"id": "maker-b/model-small",
"name": "Model Small",
"input_per_million": 0.1,
"output_per_million": 0.5
}
],
"note": "More models are available than are listed here."
}
POST /api/v1/tests
Starts a test of one prompt on several models and answers at once. Cases and checks are drafted for you if you leave them out. Models are left out, dearest first, to fit max_spend_usd. Spends credit, never more than max_spend_usd by estimate.
It takes: prompt (needed), system, context, contexts, default_context, name, cases, checks, models, max_spend_usd.
curl -sS -X POST "https://springprompt.com/api/v1/tests" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
-H "Content-Type: application/json" \
--data '{
"name": "Support replies",
"prompt": "Reply to this customer message: {{message}}",
"system": "You are a support agent for a furniture shop.",
"cases": [
{
"values": {
"message": "Where is my order?"
}
},
{
"values": {
"message": "The table arrived scratched."
}
}
],
"checks": [
"Answers the question asked",
"Is polite"
],
"models": [
"maker-a/model-large",
"maker-b/model-small"
],
"max_spend_usd": 0.5
}'
It answers 202 with test_id, status, url, how many cases and checks it will use, the models, any left_out_to_fit_limit with left_out_because (which limit, and how to raise it; null when none), and most_it_can_cost_usd.
{
"keys_removed": 0,
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"status": "queued",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"cases": 2,
"checks": 2,
"models": [
"maker-a/model-large",
"maker-b/model-small"
],
"left_out_to_fit_limit": [],
"left_out_because": null,
"most_it_can_cost_usd": 0.31,
"next": "GET /api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25 until status is done or failed. Tests usually take one to three minutes."
}
GET /api/v1/tests/{test_id}
A test's status and, when it is done, what it found. With ?wait= the request is held for up to 25 seconds while the test finishes; call it again until status is done or failed. Free.
curl -sS "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with status, url, versions, results (each model's pass rate, likely range, cost per 1,000 answers and speed) and spent_usd. When done, also text and pick (the verdict), failures and cascades.
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"versions": [
{
"id": "v1",
"note": "",
"system": "You are a support agent for a furniture shop.",
"prompt": "Reply to this customer message: {{message}}"
}
],
"results": [
{
"version": "v1",
"model": "maker-b/model-small",
"name": "Model Small",
"passed_percent": 91.7,
"checks_percent": 95.8,
"passed": 11,
"of": 12,
"likely_passed_percent": [
65,
99
],
"cost_per_1000_answers": 0.42,
"cost_per_1000_usable_answers": 0.46,
"median_seconds": 1.2,
"median_ttft_seconds": 0.48,
"errors": 0
}
],
"spent_usd": 0.07,
"text": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers.",
"pick": "maker-b/model-small",
"failures": [],
"cascades": [],
"next": "Rate some answers yourself, because passing checks isn't the same as being good."
}
GET /api/v1/tests
The account's recent Prompt tests, newest first. Free.
curl -sS "https://springprompt.com/api/v1/tests" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with tests, each with a test_id, name, status, date and link.
{
"tests": [
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"created_at": "2026-11-04T09:10:02+00:00",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b"
}
]
}
GET /api/v1/tests/{test_id}/recommendation
Which model one saved test's answers point to, read from the answers the test already holds, with what has changed since it last ran. Add ?prompt_hash=sha256:... with your own prompt's hash (see below) to get prompt_matches, true or false. Free: it runs nothing.
curl -sS "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b/recommendation" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with recommended_model (or null), whether it passes, its pass_rate and cost_per_1000_usd, the runner_up, checked_at, whether it is stale and the changes since, changed_at and previous_model if a recheck moved it, and the url. Also the test's minimum_pass_rate, passes_fewer_than_best with the best_model, the prompt_version and prompt_hash it was measured on (and prompt_matches if you sent one), and retires_on with the replacement_model if the model is being retired.
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-b/model-small",
"recommended_model_name": "Model Small",
"passes": true,
"pass_rate": 91.7,
"passed": 11,
"cases": 12,
"cost_per_1000_usd": 0.42,
"cost_per_1000_usable_usd": 0.46,
"runner_up": {
"model": "maker-a/model-large",
"name": "Model Large",
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1
},
"models_compared": 3,
"reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
"note": "Model Small passed 11 of 12 cases and Model Large passed 12 of 12. Model Small is picked because it costs less and these cases show no clear difference, but it passed fewer. Set a minimum pass rate on the test if a pick may never pass fewer.",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": null,
"passes_fewer_than_best": true,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": "2027-01-15",
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
PATCH /api/v1/tests/{test_id}
Sets the lowest pass rate a saved test may recommend a model at, from 0 to 100 percent of its cases, or null to take it off. A model below it is never recommended, and rechecks and webhooks keep to it. It answers with the recommendation as it now stands. A test that is running answers 409. Free: it runs nothing.
It takes: minimum_pass_rate (needed).
curl -sS -X PATCH "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
-H "Content-Type: application/json" \
--data '{
"minimum_pass_rate": 100
}'
It answers 200 with the recommendation, with the same fields as the call above.
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-a/model-large",
"recommended_model_name": "Model Large",
"passes": true,
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1,
"runner_up": null,
"models_compared": 3,
"reason": "Model Large passed 12 of 12 at $6.10 per 1,000 usable answers.",
"note": "",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": 100.0,
"passes_fewer_than_best": false,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": null,
"replacement_model": null,
"replacement_model_name": null,
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
GET /api/v1/recommendations
The same for every saved test on the account: the model sheet as JSON. Free: it runs nothing.
curl -sS "https://springprompt.com/api/v1/recommendations" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with recommendations, one for each saved test, each with the fields of the call above.
{
"recommendations": [
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-b/model-small",
"recommended_model_name": "Model Small",
"passes": true,
"pass_rate": 91.7,
"passed": 11,
"cases": 12,
"cost_per_1000_usd": 0.42,
"cost_per_1000_usable_usd": 0.46,
"runner_up": {
"model": "maker-a/model-large",
"name": "Model Large",
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1
},
"models_compared": 3,
"reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
"note": "Model Small passed 11 of 12 cases and Model Large passed 12 of 12. Model Small is picked because it costs less and these cases show no clear difference, but it passed fewer. Set a minimum pass rate on the test if a pick may never pass fewer.",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": null,
"passes_fewer_than_best": true,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": "2027-01-15",
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
]
}
POST /api/v1/tests/{test_id}/improve
Rewrites a finished test's prompt from its failures and your ratings, adds it as a new version and tests it on the same cases, checks and models. Spends credit: about one more run of the test.
curl -sS -X POST "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b/improve" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 202 with test_id, the new version, status, the changes made, and the rewritten prompt and system.
{
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"version": "v2",
"status": "queued",
"changes": [
"Says to answer the question in the first sentence"
],
"prompt": "Reply to this customer message: {{message}}\nAnswer the question in your first sentence.",
"system": "You are a support agent for a furniture shop.",
"next": "GET /api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b?wait=25 until status is done."
}
GET /api/v1/judges
The account's trained judges: each learned from one person's ratings and says how often it agreed with them on answers it had not seen. Free.
curl -sS "https://springprompt.com/api/v1/judges" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
It answers 200 with judges, each with an id, name and agreement.
{
"judges": [
{
"id": "judge_3f9a",
"name": "Support tone",
"labels": 60,
"agreement": 0.86,
"baseline": 0.62,
"ready": true,
"summary": "Agreed on 86% of answers it hadn't seen. Always giving the commoner verdict would get 62%. Trained on 60 ratings; a fast classifier."
}
]
}
POST /api/v1/judges/{judge_id}/score
Scores up to 50 answers with a trained judge: how likely the person it learned from is to call each one good, from 0 to 1. Spends a little credit.
It takes: answers (needed).
curl -sS -X POST "https://springprompt.com/api/v1/judges/judge_3f9a/score" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
-H "Content-Type: application/json" \
--data '{
"answers": [
"Your order left our warehouse today.",
"Check the website."
]
}'
It answers 200 with judge, its name, and good_probability, one number per answer.
{
"judge": "Support tone",
"good_probability": [
0.93,
0.12
]
}
The MCP tool behind each call has the same fields: list_models, test_prompt, get_test, list_tests, get_recommendation, set_minimum_pass_rate, list_recommendations, improve_prompt, list_judges, score_answers.
Shared context
Many apps give the model more than the prompt: a brief, a style guide, project files or retrieved documents. Send that text once as context when you start a test, up to 50,000 characters, and leave the prompt as your app has it. It is the same for every case, unlike {{variables}}.
- Every version and model gets it on every case. By default it goes first in the system message, before your own system text, as
Reference material the app gives the model:followed by the text inside<context>tags. If your prompt or system has{{context}}, the text goes there instead and nothing is added, and cases need no value for it. - Drafted cases and checks read it as what the prompt is for, so they match its audience and tone. The judge reads it as background, so a good answer means good for that brief.
- It stays with the test: improving the prompt, added versions and cases, repeats, rechecks and merge checks all use it as it is. Improving never edits it.
- It is sent with every call, so it is in the estimate and the cost. A long one leaves out more models to fit
max_spend_usd. One over the limit is refused with422, never cut short.
Several contexts
If your app sends one of several briefs or document sets, a brief per market or client say, send them as contexts instead: up to 8, each {"name": "Retail brief", "text": "…"}, each up to 50,000 characters and 100,000 in all.
- Each case gets one: the one its own
"context"names, or else the default. The default is the first, or the onedefault_contextnames, or no context at all with"none". A case can say"context": "none"too. - Every version and model gets the same context for the same case, so their answers compare. The judge reads that case's context as background.
- Drafted cases are spread across the contexts, each linked to the one it was written for. Drafted checks read all of them, each by name.
- A case that names a context the test doesn't have is refused with
422, in words. - Each case is priced with its own context's length. Plain
contextstill works, as one context every case gets.
Both work the same way for a conversation test, where the agent gets the context at the start of its system prompt and each scenario may name its own. GET /api/v1/tests/{test_id} says how long the default one is in context_characters, and lists each context's name, length and how many cases get it in contexts.
Conversation tests
A conversation test checks an agent's prompt over whole conversations instead of single answers. A simulated user talks to each model for several turns, the agent can call tools you describe, and your checks are run on each conversation from start to finish. It is a Prompt test in every other way: it has versions, a recommendation and a page in the app, and the calls above read it.
The tools in this test are dummies. Nothing real is called or changed: each call gets back the result written for it. So a test can never send an email, move money or change a record.
Start one
Send POST /api/v1/tests with "mode": "conversation". A call that sends scenarios is taken as a conversation test even without it. Nothing is drafted in this call: send at least one scenario. To have some written for you first, see Draft scenarios and tools.
| Field | What it is |
|---|---|
prompt | The agent's prompt. It is sent as the system prompt, exactly as written: {{variables}} are not filled in. If you send system too, it goes in front. |
scenarios | Up to 12. Each is one person with one thing they want: goal (needed), who they are, what they knows, their opening message (written for them if left out), the most turns allowed (default 8, at most 12), notes for the judge on what a good conversation does, and results. |
tools | Up to 8 dummy tools: name, description, parameters (a JSON schema for the arguments) and result, which is what every call gets back. Leave tools out to test a conversation with no tools. |
scenarios[].results | Up to 8 for a scenario. Each sets what one tool returns in that scenario instead of its usual result: tool, result and, if you like, args. With args, it applies only to calls whose arguments include them. This is how you test a tool error or a branch. |
checks | A plain sentence is a yes or no question the judge answers from the whole transcript. An object is a rule worked out in code, the same way every time; the kinds are listed below. Each can be a "tier": "must" (the default) or "stretch". If you send no checks, the test uses three: the user got what they wanted, no invalid tool calls, and no loop. |
user_model | One model plays the user. Leave this out for the default, or send any id GET /api/v1/models lists to choose another. It is kept with the test, so later runs meet the same user, and its calls are in the estimate. |
models | Up to 6. If you send none, a starting mix of 3. |
max_spend_usd | As for any test. A conversation is several model calls, so it costs more than a single answer. |
curl -sS -X POST "https://springprompt.com/api/v1/tests" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
-H "Content-Type: application/json" \
--data '{
"mode": "conversation",
"name": "Support agent",
"prompt": "You are the support agent for a furniture shop. Look an order up before you say anything about it. Hand over to a person when someone asks for a refund above £200.",
"tools": [
{
"name": "look_up_order",
"description": "Finds an order by its number and returns its status.",
"parameters": {
"type": "object",
"properties": {
"order": {
"type": "string"
}
},
"required": [
"order"
]
},
"result": {
"status": "shipped",
"arrives": "Thursday"
}
},
{
"name": "hand_over",
"description": "Passes the conversation to a person.",
"parameters": {
"type": "object",
"properties": {
"reason": {
"type": "string"
}
},
"required": [
"reason"
]
},
"result": {
"ok": true
}
}
],
"scenarios": [
{
"goal": "To find out where their order is",
"who": "A customer in a hurry, short with their words",
"knows": "Order number 48213.",
"turns": 6
},
{
"goal": "A refund of £350 for a table that arrived scratched",
"knows": "Order number 51907.",
"results": [
{
"tool": "look_up_order",
"args": {
"order": "51907"
},
"result": {
"status": "delivered",
"total": "£350"
}
}
]
}
],
"checks": [
{
"kind": "called",
"tool": "look_up_order"
},
{
"kind": "before",
"first": "look_up_order",
"then": "hand_over"
},
{
"kind": "valid_calls"
},
{
"kind": "no_loop"
},
{
"kind": "resolved",
"tier": "stretch"
},
"Never promises a refund above £200 itself"
],
"models": [
"maker-a/model-large",
"maker-b/model-small"
],
"max_spend_usd": 2
}'
It answers 202 with the same fields as any test, except that scenarios takes the place of cases.
{
"keys_removed": 0,
"test_id": "9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d",
"status": "queued",
"url": "https://springprompt.com/app/prompt/9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d",
"api_url": "https://springprompt.com/api/v1/tests/9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d",
"scenarios": 2,
"checks": 6,
"models": [
"maker-a/model-large",
"maker-b/model-small"
],
"left_out_to_fit_limit": [],
"left_out_because": null,
"most_it_can_cost_usd": 0.84,
"mode": "conversation",
"next": "GET /api/v1/tests/9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d?wait=25 until status is done or failed. Tests usually take one to three minutes."
}
The rules a check can be
kind | Fields | Passes when |
|---|---|---|
resolved | None | The simulated user ended the conversation saying they got what they wanted. |
called | tool, args, min, max | The tool was called: at least once, or between min and max times. With args, only calls whose arguments include them count. |
not_called | tool, args | The tool was never called, or never with arguments that include args. |
before | first, then | The first tool was called before the then tool. It does not apply when the then tool was never called. |
turns | max | The user ended the conversation within max turns. |
valid_calls | None | No call named a tool that is not defined or sent arguments its schema refuses. |
no_loop | None | The same tool was never called three times with the same arguments. |
The rules a single-turn test has, such as includes, excludes and words, work here too and read the agent's last message.
How a conversation runs
One model plays the user, the same one for every model you test, so each model meets the same person. You can choose which with user_model. It stays in character, writes one short message a turn, gives a fact only when the agent asks for it, and ends the conversation by saying whether it got what it wanted or gave up. Each scenario is sent to that model. Because the user is a model too, two runs of the same test will not be word for word the same: read a pass rate on a few scenarios as a guide.
On each turn the agent is called with your tools. Each tool call is answered with its dummy result and the agent is called again, up to 6 times a turn and 40 tool calls a conversation. A call to a tool you did not define, or with arguments its schema refuses, gets an error back and is counted as an invalid call. A conversation stops when the user ends it, at its turn limit, on an error, or when it reaches its budget, which is its share of the estimate you were shown.
We cannot tell in advance which models can call tools. A model that cannot is reported in error on its conversations, not as a failed check.
Read one
GET /api/v1/tests/{test_id} answers as for any test, with costs counted per conversation, each model's typical turns and invalid calls, and one row in conversations for each conversation. Transcripts are left out unless you add transcripts=1.
curl -sS "https://springprompt.com/api/v1/tests/9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d?wait=25&transcripts=1" -H "Authorization: Bearer $SPRINGPROMPT_KEY"
{
"test_id": "9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d",
"name": "Support agent",
"status": "done",
"url": "https://springprompt.com/app/prompt/9b1d4c22-7a3e-4f60-8c5d-1e2f3a4b5c6d",
"versions": [
{
"id": "v1",
"note": "",
"system": "",
"prompt": "You are the support agent for a furniture shop. Look an order up before you say anything about it. Hand over to a person when someone asks for a refund above £200."
}
],
"results": [
{
"version": "v1",
"model": "maker-b/model-small",
"name": "Model Small",
"passed_percent": 100.0,
"checks_percent": 100.0,
"passed": 2,
"of": 2,
"likely_passed_percent": [
34,
100
],
"median_seconds": 6.2,
"errors": 0,
"cost_per_1000_conversations": 3.1,
"cost_per_1000_usable_conversations": 3.1,
"cost_per_conversation_usd": 0.0031,
"typical_turns": 2.5,
"tool_calls": 3,
"invalid_tool_calls": 0,
"invalid_call_percent": 0.0
}
],
"spent_usd": 0.09,
"text": "Model Small passed 2 of 2 at $3.10 per 1,000 usable conversations.",
"pick": "maker-b/model-small",
"failures": [],
"next": "Read some conversations yourself, because passing checks isn't the same as being good.",
"mode": "conversation",
"unit": "conversation",
"user_model": "maker-c/model-user",
"tools": [
"look_up_order",
"hand_over"
],
"scenarios": [
{
"id": "c1",
"wants": "To find out where their order is",
"who": "A customer in a hurry, short with their words",
"turns": 6
}
],
"conversations": [
{
"version": "v1",
"model": "maker-b/model-small",
"name": "Model Small",
"scenario": "c1",
"passed": true,
"turns": 2,
"tool_calls": 1,
"invalid_tool_calls": 0,
"ended": "done",
"ended_text": "The user got what they wanted.",
"cost_usd": 0.0029,
"seconds": 6.2,
"failed_checks": [],
"error": null,
"transcript": [
{
"role": "user",
"turn": 1,
"text": "Where's my order?"
},
{
"role": "agent",
"turn": 1,
"text": "I can check that. What is your order number?",
"tool_calls": []
},
{
"role": "user",
"turn": 2,
"text": "48213"
},
{
"role": "agent",
"turn": 2,
"text": "",
"tool_calls": [
{
"tool": "look_up_order",
"arguments": "{\"order\": \"48213\"}",
"result": "{\"status\": \"shipped\", \"arrives\": \"Thursday\"}",
"invalid": false,
"error": ""
}
]
},
{
"role": "agent",
"turn": 2,
"text": "It has shipped and arrives on Thursday.",
"tool_calls": []
},
{
"role": "user",
"turn": 2,
"text": "Thanks, that's all.",
"ended": "done"
}
]
}
]
}
typical_turnsis the median number of turns in a model's conversations.invalid_call_percentis the share of its tool calls that were invalid, and null when it made none.cost_per_conversation_usdand the twocost_per_1000figures are what the model under test cost at its own prices. The model that plays the user and the judge are charged to the test, and are inspent_usd, but they are not part of a model's cost.endedis one of:done(the user got what they wanted),gave_up(the user gave up),turns(it reached the turn limit before the user had what they wanted),stuck(the agent kept calling tools without answering the user),budget(it reached its budget before it finished),error(a model call failed).failed_checkslists each check a conversation failed, with theturnit failed on where that is known and null where it is not:
{
"check": "No invalid tool calls",
"tier": "must",
"turn": 2,
"detail": "Turn 2: Invalid arguments: order is required"
}
In a transcript, arguments is the text the model sent, which may not be valid JSON, and result is the text it got back. A recommendation for a conversation test has "unit": "conversation": its cases are scenarios and its costs are per 1,000 conversations.
When the test is done, cascades may suggest letting a cheaper model hold the conversation and handing it to another when it trips a check code can watch as it goes, such as an invalid call or a loop. It is worked out from the conversations the test already holds, and its caveat says what that leaves out.
Draft scenarios and tools
POST /api/v1/conversations/draft, or the draft_conversation tool over MCP, drafts scenarios or dummy tools from an agent's prompt, as the form in the app does. Send prompt, "what": "scenarios" (the default) or "tools", and for scenarios the test's tools, so a scenario can set what one returns. Nothing is saved or run: read the draft, change what you need, and send it to POST /api/v1/tests. A draft spends a little credit and counts against the same daily allowance as drafts in the app. A prompt that names nothing the agent can call gets a 422 when you ask for tools.
curl -sS -X POST "https://springprompt.com/api/v1/conversations/draft" -H "Authorization: Bearer $SPRINGPROMPT_KEY" \
-H "Content-Type: application/json" \
--data '{
"what": "scenarios",
"prompt": "You are the support agent for a furniture shop. Look an order up before you say anything about it. Hand over to a person when someone asks for a refund above £200.",
"tools": [
{
"name": "look_up_order",
"description": "Finds an order by its number and returns its status.",
"parameters": {
"type": "object",
"properties": {
"order": {
"type": "string"
}
},
"required": [
"order"
]
},
"result": {
"status": "shipped",
"arrives": "Thursday"
}
},
{
"name": "hand_over",
"description": "Passes the conversation to a person.",
"parameters": {
"type": "object",
"properties": {
"reason": {
"type": "string"
}
},
"required": [
"reason"
]
},
"result": {
"ok": true
}
}
]
}'
{
"scenarios": [
{
"goal": "A refund of £320 for a sofa that arrived torn",
"who": "A calm customer who writes in full sentences",
"knows": "Order number 66120.",
"opening": "Hello, my sofa arrived torn and I'd like my money back.",
"turns": 8,
"notes": "Looks the order up, then hands over: the refund is above £200.",
"results": [
{
"tool": "look_up_order",
"args": {},
"result": "{\"status\": \"delivered\", \"total\": \"£320\"}"
}
]
}
]
}
Conversation tests can also be checked before a merge, with each scenario counting as a case.
What “passes” and “recommended” mean
A recommendation is read from the answers a saved test already holds. Asking for one runs no model and spends no credit, so you can ask as often as you like. It is only as good as the test: a handful of cases cannot tell two close models apart, and it says so in note.
- An answer passes when it came back without an error and met every must check of the test. A case that was run several times passes if it passed in most of its runs.
- pass_rate is the share of the test's cases that passed, from 0 to 100, on the newest version of the prompt that has answers. passes is true when a model is recommended, and false when there is none to recommend; it does not mean every case passed, so read pass_rate for that.
- recommended_model is the test's own pick, the one named at the top of its page: the model that passed the most cases (the cheapest of them if several tie), unless a model that costs less per usable answer shows no clear difference from it on these cases. Then it is the cheapest such model. “No clear difference” is an exact sign test on the cases where the two differ, and it is only applied with 6 cases or more; with fewer, the pick is simply the model that passed the most.
- Cheapest means cost_per_1000_usable_usd: the cost of 1,000 answers divided by the pass rate, since failed answers are paid for too.
- A model that passed no case is never recommended, and a test with no must check has no recommendation: recommended_model is null and note says why.
- runner_up is what the same rule picks once the recommended model is left out.
- A test can have a minimum pass rate (minimum_pass_rate, from 0 to 100, null for none), set on the test's page or with PATCH /tests/{test_id}. A model whose share of passed cases is below it is never recommended or named runner-up, and the rule above picks from the rest. If none reaches it, recommended_model is null and note says so, with the best model's figures. The share is worked out from the counts, so 7 of 9 (77.78%) is below a minimum of 77.8. A test with no minimum is exactly as above.
- passes_fewer_than_best is true when the recommended model passed fewer cases than the best-passing model, whatever the minimum. That can happen when a cheaper model shows no clear difference on few cases. best_model, best_model_name, best_passed, best_cases and best_pass_rate are that model's, and note says it in words. A job that applies changes by itself can refuse a pick with passes_fewer_than_best, or the test can set a minimum so such a pick is never made.
stale is true when something has changed since the test last ran: a newer model in a line the test uses, a changed list price, or a model that is being retired. changes says what. The answers are then older than the market, and a rerun would say whether the recommendation still holds. changed_at and previous_model are set when a scheduled recheck moved the recommendation, and are null otherwise.
Every field above keeps its name and meaning. New fields may be added, so ignore what your code does not know.
Benchmarks on this site compare models on fixed public tasks. A recommendation here is narrower: it is about your prompt and your cases, and it holds only for them.
What changed: the kinds
Each entry in changes has a kind, and so does each change email. It is always one of these, and a new kind would be added to this list first:
| kind | What it means |
|---|---|
retirement | A model in the test has a retirement date from its developer, and the replacement the developer names has not been run in this test. It is flagged as soon as the retirement is known to us, not on the day, and for 30 days after the date. |
launch | Since the test last ran, a newer model has come out in the same family as one in the test, or a notable new model from a main lab, and the test has not been run on it. |
price | The list price of a model in the test has moved by more than 1% since the test last ran, so its costs and the cheapest model that passes may be out of date. |
A recommendation.changed event has its own kind, one of model, retirement: see the webhooks.
Retirement dates
When the recommended model has a retirement date from its developer, retires_on is that date (YYYY-MM-DD) and replacement_model and replacement_model_name are the replacement the developer names. Otherwise all three are null. retirements lists every model in the test that has one, soonest first, each with model, retires_on, retired (true once the date has passed), the replacement, the developer and a page on this site with the developer's notice. If the model you run now is in the test, look for it there. A retirement is shown as soon as we know of it, usually months ahead, and for 30 days after the date. Every retirement we track is listed with its source.
Which prompt a result is for
prompt_version is the id of the prompt version the recommendation was measured on (the newest with answers, the same as version), and prompt_hash is a hash of exactly what the models were given for it. Both are null when the test has no answers. Compute the same hash from the prompt your app runs, and if the two differ, the recommendation was measured on another prompt.
The hash covers three things and nothing else: the version's system text, its prompt template with the {{variables}} left as written, and the text of each of the test's shared contexts, in the test's order. The system text and prompt are exactly versions[].system and versions[].prompt as GET /api/v1/tests/{test_id} shows them. Each context is the text you sent, with spaces and line breaks at its start and end removed, as we store it (the system text and prompt are not trimmed). Cases, checks, models, context names and version notes are not in it.
- Make a JSON object with three keys:
"contexts", a list of strings (empty,[], when the test has no shared context);"prompt", a string; and"system", a string (""when there is none). - Write it with the keys in that order (sorted), no spaces or line breaks between tokens, and characters outside ASCII written as they are, not as
\uescapes. The strings are taken exactly as they are, with no trimming. - Take the SHA-256 of those bytes in UTF-8, in lower-case hex, and put
sha256:in front.
For example, this system text, prompt and one context:
{
"contexts": [
"Returns are free within 30 days. Delivery takes 3\u20135 working days."
],
"prompt": "Reply to this customer message: {{message}}",
"system": "You are a support agent for a furniture shop."
}
are written as these bytes:
{"contexts":["Returns are free within 30 days. Delivery takes 3–5 working days."],"prompt":"Reply to this customer message: {{message}}","system":"You are a support agent for a furniture shop."}
and hash to sha256:1c37e49c7c5f41c57829228c5dc9ab16902f413f0fb2918b4e38e1892f404ad6. In Python:
import hashlib, json
def prompt_hash(system: str, prompt: str, contexts: list[str]) -> str:
fields = {"contexts": [c.strip() for c in contexts], "prompt": prompt, "system": system}
canonical = json.dumps(fields, sort_keys=True, separators=(",", ":"), ensure_ascii=False)
return "sha256:" + hashlib.sha256(canonical.encode("utf-8")).hexdigest()
In Node:
const crypto = require("node:crypto");
function promptHash(system, prompt, contexts) {
// The keys in this order: contexts, prompt, system. Contexts are trimmed, as we store them.
const canonical = JSON.stringify({ contexts: contexts.map((c) => c.trim()), prompt, system });
return "sha256:" + crypto.createHash("sha256").update(canonical, "utf8").digest("hex");
}
To have the server compare for you, send yours with the read: GET /api/v1/tests/{test_id}/recommendation?prompt_hash=sha256:…. prompt_matches is then true or false; it is null when you send none.
Example: a job that keeps a per-feature model list up to date
Say each feature of your product has its own saved test, named after the feature, and your code reads the model for each feature from a file. This script writes that file from your tests' recommendations. It runs nothing and spends nothing, so it is safe to run every hour.
#!/bin/sh
# Writes models.json: {"Support replies": "maker-b/model-small", ...}
set -eu
out=$(curl -sS --fail "https://springprompt.com/api/v1/recommendations" \
-H "Authorization: Bearer $SPRINGPROMPT_KEY")
echo "$out" | jq '[.recommendations[]
| select(.recommended_model != null)
| {key: .name, value: .recommended_model}] | from_entries' > models.json.new
mv models.json.new models.json
# Tests whose answers are older than a change in the market:
echo "$out" | jq -r '.recommendations[] | select(.stale) | "\(.name): \(.changes[0].text)"'
Run it from cron, for example at ten past every hour:
10 * * * * SPRINGPROMPT_KEY=... /opt/app/bin/update-models.sh
The script keeps the old file if a call fails, and leaves out a test that has no recommendation. A saved test is deleted 90 days after it was made, and its recommendation goes with it, so as written the script would then drop that feature from the file. A test with scheduled rechecks on is the exception: it is kept for as long as they stay on, and for 90 days after they go off. For a test your job depends on, switch them on, make the test again before it goes, or have your job keep the last model it knew. Have a person look at a change before it reaches production if the feature matters: previous_model and changed_at tell you when a recommendation moved, and reason says why the model was picked. A job that applies changes by itself can also skip a pick whose passes_fewer_than_best is true (or set a minimum pass rate so there is none), check prompt_hash against the prompt it runs, and plan for a retires_on date.
The recommendations only move when the tests are run again. You can rerun a test from its page when it is flagged, or switch on a scheduled recheck for it on the model sheet in the app: it reruns the test when something changes and once a week, within a spending limit (at the moment at most $25.00 a recheck and $100.00 on one account in any 24 hours, unless you set your own for your account under Spending on the credit page), and stops if nobody has looked at the test for 3 rechecks in a row.
Webhooks
Instead of asking again and again, you can have us tell your server. Set one https address under Webhook on the keys page, and we post a small signed JSON event to it when a run of one of your tests ends, or when a run changes the model a test recommends. An account has one webhook. A webhook needs the Pro plan, as calling the API does.
Setting the address makes a signing secret, which is shown once. On the same page you can send a test event (its type is webhook.test), switch the webhook off and on, make a new secret, and see the last deliveries with the status your server answered.
The address has to start with https://, carry no user name or password, and belong to a host on the public internet. We refuse a host at a private, loopback or link-local address, check again every time we send, and do not follow redirects.
The events
| Type | Sent when |
|---|---|
test.finished | A run of one of your Prompt tests has ended, done or failed, whoever started it: you in the app, a key over the API, a coding agent over MCP or a scheduled recheck. |
recommendation.changed | A run has ended and the test now recommends a different model from the one it recommended before that run (kind model). It is sent after the test.finished event for the same run. A test's first run does not send it, and neither does a run that failed. Also once per test for each retirement of the model it recommends (kind retirement). |
The body is a JSON object. status is done or failed. started_by says what started the run: app, api, mcp, schedule. recommendation is exactly what GET /api/v1/tests/{id}/recommendation answers. As everywhere on this page, the model ids and the figures in the examples are made up.
An event never carries your prompts, test cases or answers. Treat it as a hint that something is ready: fetch the test from api_url with your key, and act on what that call answers.
The kinds of recommendation.changed
A recommendation.changed event has a kind, always one of these. Events sent before kind was added were all of the first kind.
| kind | What it means |
|---|---|
model | The test now recommends a different model from the one it recommended before a run: previous_model is the old one, recommendation.recommended_model the new one (null when no model is recommended any more, for example none reaches the test's minimum pass rate). retirement is null. |
retirement | The model the test recommends, or the one it recommended before a scheduled recheck moved it, has a retirement date. Nothing about the recommendation moved: previous_model is the same as recommendation.recommended_model. retirement says which model, retires_on (an ISO date) and the replacement_model its developer names. |
A job that applies a change by itself should act only on kind model. When the test has a minimum pass rate, no event names a recommended model below it.
A retirement is sent once for each test and each retirement, the first time our hourly pass sees it: as soon as the retirement is known to us, which is usually months ahead, and not again as the date nears. It is about the model the test recommends, or the one it recommended before a scheduled recheck moved it. A retirement of another model in the test is not sent, but it is in recommendation.retirements. Rerun the test with the replacement to see how it does.
recommendation.changed with kind retirement:
{
"id": "evt_5c4d3e2f1a0b9c8d7e6f5a4b3c2d1e0f",
"type": "recommendation.changed",
"created_at": "2026-11-04T09:12:45+00:00",
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"test_name": "Support replies",
"status": "done",
"started_by": null,
"api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"previous_model": "maker-b/model-small",
"kind": "retirement",
"retirement": {
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
},
"recommendation": {
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-b/model-small",
"recommended_model_name": "Model Small",
"passes": true,
"pass_rate": 91.7,
"passed": 11,
"cases": 12,
"cost_per_1000_usd": 0.42,
"cost_per_1000_usable_usd": 0.46,
"runner_up": {
"model": "maker-a/model-large",
"name": "Model Large",
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1
},
"models_compared": 3,
"reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
"note": "Model Small passed 11 of 12 cases and Model Large passed 12 of 12. Model Small is picked because it costs less and these cases show no clear difference, but it passed fewer. Set a minimum pass rate on the test if a pick may never pass fewer.",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": null,
"passes_fewer_than_best": true,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": "2027-01-15",
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
}
test.finished
{
"id": "evt_3b1f0c9d5e7a4f2c8d6b0a1e2f3c4d5e",
"type": "test.finished",
"created_at": "2026-11-04T09:12:45+00:00",
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"test_name": "Support replies",
"status": "done",
"started_by": "api",
"api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"recommendation": {
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-b/model-small",
"recommended_model_name": "Model Small",
"passes": true,
"pass_rate": 91.7,
"passed": 11,
"cases": 12,
"cost_per_1000_usd": 0.42,
"cost_per_1000_usable_usd": 0.46,
"runner_up": {
"model": "maker-a/model-large",
"name": "Model Large",
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1
},
"models_compared": 3,
"reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
"note": "Model Small passed 11 of 12 cases and Model Large passed 12 of 12. Model Small is picked because it costs less and these cases show no clear difference, but it passed fewer. Set a minimum pass rate on the test if a pick may never pass fewer.",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": null,
"passes_fewer_than_best": true,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": "2027-01-15",
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
}
recommendation.changed
{
"id": "evt_9a8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d",
"type": "recommendation.changed",
"created_at": "2026-11-04T09:12:45+00:00",
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"test_name": "Support replies",
"status": "done",
"started_by": "schedule",
"api_url": "https://springprompt.com/api/v1/tests/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"previous_model": "maker-a/model-large",
"kind": "model",
"retirement": null,
"recommendation": {
"test_id": "7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"name": "Support replies",
"status": "done",
"version": "v2",
"recommended_model": "maker-b/model-small",
"recommended_model_name": "Model Small",
"passes": true,
"pass_rate": 91.7,
"passed": 11,
"cases": 12,
"cost_per_1000_usd": 0.42,
"cost_per_1000_usable_usd": 0.46,
"runner_up": {
"model": "maker-a/model-large",
"name": "Model Large",
"pass_rate": 100.0,
"passed": 12,
"cases": 12,
"cost_per_1000_usd": 6.1,
"cost_per_1000_usable_usd": 6.1
},
"models_compared": 3,
"reason": "Model Small passed 11 of 12 at $0.46 per 1,000 usable answers. Model Large passed 12 of 12, which is no clear difference on these cases, at $6.10.",
"note": "Model Small passed 11 of 12 cases and Model Large passed 12 of 12. Model Small is picked because it costs less and these cases show no clear difference, but it passed fewer. Set a minimum pass rate on the test if a pick may never pass fewer.",
"checked_at": "2026-11-04T09:12:44+00:00",
"stale": true,
"changes": [
{
"kind": "price",
"text": "Model Small's list price has gone down since this test last ran."
}
],
"changed_at": "2026-11-04T09:12:44+00:00",
"previous_model": "maker-a/model-large",
"recheck": "on",
"url": "https://springprompt.com/app/prompt/7f3c2a10-5b6d-4e8f-9a1b-2c3d4e5f6a7b",
"minimum_pass_rate": null,
"passes_fewer_than_best": true,
"best_model": "maker-a/model-large",
"best_model_name": "Model Large",
"best_passed": 12,
"best_cases": 12,
"best_pass_rate": 100.0,
"prompt_version": "v2",
"prompt_hash": "sha256:a7c086039a24ef222e878a97643f120ea98c9c4e963e77d5d75e5419552ce957",
"prompt_matches": null,
"retires_on": "2027-01-15",
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"retirements": [
{
"model": "maker-b/model-small",
"name": "Model Small",
"retires_on": "2027-01-15",
"retired": false,
"replacement_model": "maker-b/model-small-2",
"replacement_model_name": "Model Small 2",
"developer": "Maker B",
"retirement": "model-small",
"page": "/retiring/model-small"
}
]
}
}
An event never carries your prompt, your cases or any answer. Treat it as a hint that something changed, and fetch api_url with your key for the rest. We may add fields and event types later, so ignore what your code does not know.
Check the signature
Each event comes with these headers:
X-SpringPrompt-Event | The event's type, the same as type in the body. |
|---|---|
X-SpringPrompt-Delivery | The delivery's id. It stays the same each time we try the same event again. |
X-SpringPrompt-Signature | t=<unix seconds>,v1=<signature>, made new for each attempt. |
The signature is the HMAC-SHA256, in hex, of the time, a full stop and the raw body, made with your signing secret as the key. Work it out over the bytes you received, before you parse the JSON, compare it without leaking timing, and refuse an event whose time is more than 5 minutes from now. In Python:
import hashlib, hmac, time
def verified(secret: str, raw_body: bytes, header: str) -> bool:
parts = dict(p.split("=", 1) for p in header.split(",") if "=" in p)
try:
at = int(parts.get("t", ""))
except ValueError:
return False
if abs(time.time() - at) > 300: # too old, or a clock far out
return False
signed = f"{at}.".encode() + raw_body
want = hmac.new(secret.encode(), signed, hashlib.sha256).hexdigest()
return hmac.compare_digest(want, parts.get("v1", ""))
In Node:
const crypto = require("node:crypto");
function verified(secret, rawBody, header) {
const parts = Object.fromEntries(header.split(",").map((p) => p.split("=")));
const at = Number(parts.t);
if (!Number.isInteger(at) || Math.abs(Date.now() / 1000 - at) > 300) return false;
const want = crypto.createHmac("sha256", secret).update(`${at}.`).update(rawBody).digest("hex");
const got = Buffer.from(String(parts.v1 || ""));
return got.length === want.length && crypto.timingSafeEqual(got, Buffer.from(want));
}
Keep the secret in your server's secrets. If it may have leaked, make a new one on the keys page: the old one stops signing at once.
Delivery, retries and repeats
- Answer with any
2xxstatus within 10 seconds. Answer first and do the work afterwards: put the event on a queue, or note the test's id and fetch it later. - Any other answer, a redirect included, or no answer in time counts as a failure. We then send the event again after 1 minute, 5 minutes, 30 minutes, 2 hours and 6 hours: 6 attempts in all. After the last one we give up on that event.
- The same event can reach you more than once, for example when your answer was lost on the way.
X-SpringPrompt-Deliveryand theidin the body stay the same each time, so keep the ids you have handled and skip the repeats. Events can also arrive out of order. - When 15 events in a row have been given up, we switch the webhook off and say why on the keys page. Events from the time it was off are not sent later.
- Polling still works exactly as before, and it is the fallback:
GET /api/v1/tests/{id}?wait=25for one test andGET /api/v1/recommendationsfor all of them. A webhook is a hint, not a guarantee, so a slow job like the one above is a sensible safety net for an event that never arrived. - We read no more than the first few kilobytes of your answer and keep none of it. The keys page lists your deliveries for 30 days, without the events' contents.
Checks from a pull request
Two more calls take the same keys and sit outside /api/v1: POST /api/checks runs a changed prompt or a different model against a saved test and answers pass, review or fail, and GET /api/checks/{id} reads the verdict. They have their own spending limits (at the moment $10.00 a check and $100.00 on one account in any 24 hours). Check a prompt or model change before you merge has a workflow to copy, the fields and what the verdict means.
Limits
Reading is free. Starting a test, improving a prompt and scoring answers spend the account's credit on the model calls they make. Before anything runs, the server works out the most the call could cost and refuses it if that is over a limit or over the credit left, so a refused call spends nothing. You are charged what a test really cost, which a finished test reports as spent_usd.
| Credit one key may use | $100.00 in any 24 hours across calls to /api/v1, unless you set the key its own limit on the keys page (at most $1,000.00). A call is checked against the most it could cost before anything runs. |
|---|---|
| Credit for one test | $50 unless the call sets max_spend_usd, and never more than $1,000. You can set your own default and most for your account under Spending on the credit page, from $0 up to $1,000. Need more than $1,000? Email hello@springprompt.com. |
| Calls | 2,000 a day for each account, shared with the MCP tools. |
| Waiting on one read | Up to 25 seconds. |
| Size of one request | 1 MB. |
| Models in one test | Up to 6. |
| Cases in one test | Up to 50. |
| Keys | Up to 10 in use on one account. |
Never put API keys, passwords or real customer data in a prompt or a case. A prompt, its shared context and its cases are sent to the models you choose and to the model that marks the answers. If a key is found in what you send, it is removed before the test is saved, and keys_removed says how many were.
Errors
An error is a JSON object with one field, {"error": "a plain reason"}, and one of these statuses. Nothing is spent on a call that is refused.
400 | The body was not a JSON object. |
|---|---|
401 | No key, or a key that has been removed. |
402 | The account's plan does not include what was asked (the X-Plan-Needed header names the plan that does), or there is not enough credit for it. |
404 | No such route, or no such test or judge on this account. |
409 | The test is running. Wait for it to finish. |
413 | The request was larger than the size limit. |
422 | A field is missing or cannot be used. The reason says which. |
429 | The day's calls are used up, or the call could take the key past its spending limit for 24 hours. Nothing was spent. |
A test that could not finish is not an error of the call: reading it answers 200 with "status": "failed" and the reason in error, and nothing is charged for it.
What we keep
For each call we record, against your account, which endpoint was called and when, and if it was refused, why. Credit a call spends is recorded with the key that spent it, which is how a key's limit is kept. We do not keep the body of a request beyond what the action itself saves: a test you start is saved to your account like any other test, with its prompt, shared context, cases and answers. If you set a webhook, we store its address and signing secret, and keep a list of what we sent it for 30 days, without the events' contents. The privacy page has the full list.