Docs Build with
Check a prompt or model change before you merge
A saved Prompt test holds your test cases, your checks and the answers your current prompt gave. This check runs a candidate (a changed prompt, a changed system message, a different model, or any mix) on the same cases with the same checks, compares it with those saved answers, and answers pass, review or fail. Our GitHub Action does it on every pull request: it finds the prompts the pull request rewords, checks each one, and comments with the verdict and the answers before and after.
What you need
- A Spring Prompt account. A free trial account is enough to try it. No credit card needed. Start one here.
- Prompts with a test to hold changes to. If your app sends its calls to us with Spring Prompt Live (
npx springprompt init), each prompt we have tested has one: the first look tests your first three, and you can test any other from its page. Or use any saved Prompt test. - An account key from Connect your agent in the app (the same keys the MCP server takes). Keep it in your CI's secrets, never in the repo.
The GitHub Action
Save this as .github/workflows/prompt-check.yml and add your key as the repository secret SPRINGPROMPT_KEY. Nothing else to set: on every pull request it sends the diff, we find each tested prompt whose wording it changes, and check the new wording. The job fails on fail, warns on review, and the result is posted as a comment on the pull request, updated on each push.
name: Prompt check
on: pull_request
permissions:
contents: read
pull-requests: write
jobs:
check:
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 2
- uses: incremento/springprompt-action@v1
with:
key: ${{ secrets.SPRINGPROMPT_KEY }}
Options, all optional: test-ids holds the diff against saved Prompt tests as well (their ids, comma separated); fail-on: review fails the job on review too; max-spend-usd caps one check; comment: false leaves the pull request alone and writes only the job summary. The action's code is on GitHub.
How it finds your prompts
Your prompts live in code, with quotes, indentation and your own way of filling in values ({name}, ${name}), where we hold them as {{customer}}. So we do not look for prompts in the diff. Each change in it (these words became those) is looked for in the wording of every prompt you have tested, with a little unchanged text either side, ignoring spacing. Where it is found exactly once, the change is made there and that prompt is checked. Code changes match no prompt and are left alone. A prompt rewritten from scratch will not be found: check it from the test's page instead. When a file your app calls a prompt from changes but no change matches its wording, the comment says so.
What is sent: the pull request's diff (lock files and minified files left out, at most about 1.5 MB). We use it to find the changes and keep none of it.
What the comment shows
For each prompt: a card with the verdict, the share of cases passing every must check before and after, and the side-by-side split; a link into the app with every case; why it got worse and suggested fixes, on a review or a fail; each rubric before and after; the checks that now fail; the wording before and after; and up to three cases with both answers next to each other, the worst first.
One prompt file, with curl
Save this as .github/workflows/prompt-check.yml, add your key as the repository secret SPRINGPROMPT_KEY, and set the test id and the path to a prompt file that holds the prompt as written in the test. It starts a check, waits for the verdict, and fails the job on fail.
name: Prompt check
on:
pull_request:
paths:
- "prompts/support.txt"
jobs:
check:
runs-on: ubuntu-latest
timeout-minutes: 25
steps:
- uses: actions/checkout@v4
- name: Check the prompt against the saved test
env:
SPRINGPROMPT_KEY: ${{ secrets.SPRINGPROMPT_KEY }}
TEST_ID: 00000000-0000-0000-0000-000000000000
PROMPT_FILE: prompts/support.txt
LABEL: "PR #${{ github.event.pull_request.number }}"
API: https://springprompt.com/api/checks
run: |
set -euo pipefail
auth="Authorization: Bearer $SPRINGPROMPT_KEY"
body=$(jq -n --arg test "$TEST_ID" --arg label "$LABEL" --rawfile prompt "$PROMPT_FILE" \
'{test_id: $test, prompt: $prompt, label: $label}')
started=$(curl -sS -X POST "$API" -H "$auth" -H "Content-Type: application/json" --data "$body")
id=$(echo "$started" | jq -r '.id // empty' 2>/dev/null || true)
if [ -z "$id" ]; then
echo "::error::The check did not start: $started"
exit 1
fi
result="$started"
for _ in $(seq 1 60); do
[ "$(echo "$result" | jq -r '.status' 2>/dev/null || echo running)" = "running" ] || break
sleep 2
result=$(curl -sS "$API/$id?wait=20" -H "$auth" || echo '{"status":"running"}')
done
echo "$result" | jq '{status, verdict, reasons, numbers, cost_usd, url}'
summary=$(echo "$result" | jq -r '.summary // .error // "It was still running when this job stopped waiting."')
url=$(echo "$result" | jq -r '.url // empty')
case "$(echo "$result" | jq -r '.verdict // empty')" in
pass) echo "Passed. $summary $url" ;;
review) echo "::warning::Review before you merge. $summary $url" ;;
fail) echo "::error::$summary $url"; exit 1 ;;
*) echo "::error::No verdict. $summary Run the job again."; exit 1 ;;
esac
To check a model change instead, send "model": "the-new-model-id" in place of (or as well as) prompt. A model id is the maker and the model's name, such as maker/model-name. The list_models tool on our MCP server lists them, a check's answer names the ids it used, and a request with an id we do not know is refused with the closest ones.
Pull requests from forks cannot read repository secrets on GitHub, so the job will not start a check for them.
What the verdict means
fail: a must check that passed on the saved answers fails on the candidate, on any case. Or quality dropped by more than chance would explain, on the checks or side by side.review: something got worse, but by an amount that could be chance on this many cases. Or part of the candidate could not be answered or judged. The job above passes with a warning; a person should read the cases that changed.pass: neither of those.
Quality is counted case by case, two ways.
- On the checks. A case is worse when the candidate passes fewer of its checks than the saved answer did (must and stretch checks alike, and a rubric by its level, not just pass or fail), and better when it passes more.
- Side by side. Checks only see what they name, so each new answer is also put next to its saved one and a judge says which does the job better for the person who will read it, or that they are as good. The judge sees what that person sent, not your system instructions: those are what changed, and your checks already hold your own rules. A judge tends to favour whichever answer it reads first, so each pair is asked twice, in both orders, and only a verdict given both ways counts. Anything else is a tie.
Each split of worse against better cases is put to an exact sign test: a drop that chance would produce less than one time in twenty is a fail, and a smaller one is a review. So a few cases moving one way, which happens on any rerun, never fail a pull request by themselves.
On a review or a fail, a model reads the checks that failed and the answers that got worse, says in a sentence or two why, and suggests up to three small edits to the new wording that keep what you were trying to do.
The answer also carries the numbers (cases compared, how many pass every must check before and after, cases better and worse, the side-by-side split, each rubric's average level before and after), the checks that got worse, the suggested fixes, what the check cost, and a link to the test's page in the app, where the check is listed with every case. The link opens for the account's owner once signed in.
The limits, plainly
- It costs credit. A check runs every case once on one model and marks the answers, then a judge compares each pair twice. So it costs a little more than running your test on that one model, plus one short call for the suggested fixes when something got worse. Starting a check tells you the most it could cost (
most_it_can_cost_usd) and sets that aside; you are charged what it really cost. If it could cost more than the credit you have left, it is refused and nothing is spent. - There are spending limits on our side. At the moment one check can use up to $10.00 of credit, and all the checks on one account up to $100.00 in any 24 hours. A check that would go over is refused before it runs. You can set your own limits for your account under Spending on your credit page, up to $1,000 each, and a lower limit for one check with
max_spend_usd. Need more than $1,000? Email hello@springprompt.com. - It takes as long as a test run. Usually one to three minutes; longer with many cases or a slow model. A check still going after 15 minutes is stopped with no verdict and no charge.
- A small test cannot say much. With fewer than six cases that changed, a drop in quality is only ever a
review. A must check that newly fails is afailon a test of any size. - Models and judges vary. The same prompt can give a different answer on a second run, and a check that a model judges can come out differently on the same answer. So a
failon a judged must check is sometimes a false alarm: run the job again before you rewrite anything. Checks that are rules (length, a phrase, valid JSON) do not vary. - The baseline is saved, not rerun. The candidate is compared with the answers your test holds for its newest version. If those are old, the model behind them may have changed since. When you merge a change, add the new prompt as a version of the test in the app, so the next check is held to it. A test holds five versions; after that, start a new test from your best version and change the id in the workflow.
- A deploy can interrupt a check. If our server restarts while a check is running, the check ends as
interruptedwith no verdict, and the credit set aside is returned. The job above fails with “No verdict”; run it again. - What we keep. The verdict, the numbers and the checks that got worse (with a short excerpt of the case and the reason) stay on the test, for the newest 30 checks. The candidate's prompt is not kept; short excerpts of up to three of its answers, next to the saved ones, are, so the pull request can show them, and so are the suggested fixes. As with any test, the prompt and cases are sent to the model you choose and to the judging model.
Conversation tests
A conversation test can be checked the same way. The candidate holds every scenario again, with the same dummy tools, the same checks and the same model playing the user, and each scenario counts as a case does here. Two things differ. A conversation costs more than one answer, so a check on a leading model can pass the limit for one check; a cheaper model, or fewer scenarios, brings it under. And the simulated user writes new messages each time, so a conversation can go another way for reasons that are not the candidate's. So when a must check newly fails on just one scenario, the result is review, not fail: look at that conversation before you merge. On two scenarios or more it is fail.
The calls
All three take the account key as a bearer token: Authorization: Bearer YOUR_KEY.
POST /api/checks
Starts a check. Send JSON:
{
"test_id": "the saved test's id",
"prompt": "the candidate prompt, with the same {{variables}} as the test",
"system": "the candidate system message",
"model": "a different model id",
"baseline_model": "which of the test's models to compare with, if it has several",
"max_spend_usd": 0.5,
"label": "PR #12"
}
Only test_id and one of prompt, system or model are needed. What you leave out stays as it is in the test. It answers 202 at once with the check's id, a poll address and most_it_can_cost_usd. If the candidate is the same as the saved prompt and model, it answers 200 with pass and runs nothing.
Refusals come back as {"error": "..."} with a plain reason: 401 without a valid key, 404 for a test that is not on the account, 409 while the test itself is running, 422 for a request that cannot be checked or would go over the limit for one check, 402 when credit is short, 429 at the daily limit, 503 if checks are switched off.
GET /api/checks/{id}
The check's state. Add ?wait=20 and it holds the request for up to 25 seconds while the check finishes. status is running, done, failed or interrupted; only done has a verdict.
{
"id": "…",
"status": "done",
"verdict": "fail",
"summary": "1 must check that passed on the baseline failed on the candidate, on 1 case.",
"reasons": ["…"],
"numbers": {"cases": 12, "baseline_passed": 11, "candidate_passed": 10,
"cases_better": 0, "cases_worse": 1, "cases_same": 11, "p": 1.0,
"must_regressions": 1, "could_not_judge": 0, "unanswered": 0, "not_compared": 0},
"regressions": [{"case": 4, "input": "…", "check": "…", "was": "pass", "now": "fail", "detail": "…"}],
"examples": [{"case": 4, "input": "…", "change": "worse", "before": "…", "after": "…", "cut": false}],
"cost_usd": 0.04,
"most_it_can_cost_usd": 0.31,
"url": "https://springprompt.com/app/prompt/…?check=…#checks"
}
The numbers above show the shape of an answer; they are not from a real test. examples holds up to three cases with the saved answer (before) and the candidate's (after), cut to 600 characters (cut says so).
POST /api/checks/diff
What the action calls. Send the output of git diff and it starts a check for each tested prompt whose wording the diff changes, at most 5:
{
"diff": "diff --git a/app/reply.py b/app/reply.py\n…",
"test_ids": ["saved tests to hold it against as well (optional)"],
"max_spend_usd": 0.5,
"label": "PR #12"
}
It answers 202 with checks (each as POST /api/checks answers, plus the prompt's name, the files its changes came from and a diff of its wording), skipped (prompts that changed but could not be checked, each with its reason), unplaced and scanned. When no tested prompt changed it answers 200 with no checks, and nothing is spent. Poll each check with GET /api/checks/{id}.
From a coding agent
The same check is on our MCP server as two tools, check_change and get_check, with the same fields. Connect your agent to use them.
From your own code
The same key also starts tests, reads results and says which model each saved test points to, as plain JSON. The API docs list every call.