Docs/Agents

Verify fixes

Checkend checks in production that a fix worked. An agent links its fix to the issue, your CI reports each deploy, and Checkend watches the error. When the error stops after the fix is deployed, the issue is verified. When the error comes back in a release that contains the fix, the issue is regressed.

The protocol is an open spec: Fix verification.

How it works#

An issue goes through these statuses:

1. unresolved (or regressed): the error happens.
2. fix_pending: an agent or a person links a fix (a commit or a pull request).
3. A deploy contains the fix. The quiet period starts. The status stays fix_pending.
4. verified: the issue had no event for the quiet period (24 hours by default).
5. regressed: the error came back in a release that contains the fix.

1Link the fix#

With MCP, the agent calls link_fix with issue_id and a commit or a pr_url. With the API:

curl -X POST "$CHECKEND_URL/v1/issues/42/fix" \
  -H "Authorization: Bearer $CHECKEND_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"commit": "a1b2c3d4e5f6"}'

The body has exactly one of these fields:

• commit: a commit SHA, 7 to 40 hex characters. A short SHA matches the full SHA.
• pr_url: a GitHub pull request URL, https://github.com/<owner>/<repo>/pull/<number>. When the project has a repository, the pull request must be in that repository.

The token needs the issues:write scope. The status becomes fix_pending. A new link replaces the open fix. When a deploy with this commit or pull request was already reported, the fix counts as deployed at once. See POST /v1/issues/:id/fix.

2Report deploys#

After each deploy, your CI sends POST /v1/deploys with a token that has the releases:write scope:

# The commits since the last deploy, as a JSON list.
COMMITS=$(git rev-list "$PREVIOUS_DEPLOY_SHA..HEAD" | jq -R . | jq -s -c .)

curl -X POST "$CHECKEND_URL/v1/deploys" \
  -H "Authorization: Bearer $CHECKEND_DEPLOY_TOKEN" \
  -H "Content-Type: application/json" \
  -d "{\"project\": \"my-app\", \"commit\": \"$(git rev-parse HEAD)\",
       \"release\": \"my-app@1.4.1\", \"environment\": \"production\",
       \"commits\": $COMMITS}"

Create the deploy token once, in the web UI: Settings → Tokens (an owner only). Give it a name, the scope releases:write, and the app. Copy the token: Checkend shows it only once. Add it to the CI secret CHECKEND_DEPLOY_TOKEN. A token for one app reports deploys and uploads source maps only for that app.

On a self-host install, a script can also create it with the admin token:

curl -X POST "$CHECKEND_URL/v1/tokens" \
  -H "Authorization: Bearer $CHECKEND_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"name": "ci-deploys", "scopes": ["releases:write"]}'

See POST /v1/deploys for all fields. On GitHub, you can use the GitHub webhook instead.

Or use the GitHub webhook#

With the GitHub webhook, GitHub tells Checkend about pushes, merged pull requests and deployments. Then your CI does not need to send the commits list.

Set the repository of the project#

curl -X PATCH "$CHECKEND_URL/v1/projects/my-app" \
  -H "Authorization: Bearer $CHECKEND_ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"repository": "my-org/my-app"}'

See PATCH /v1/projects/:project.

Connect GitHub#

1. Make a random secret, and set it as a Worker secret:

   TerminalCopy

   npx wrangler secret put GITHUB_WEBHOOK_SECRET --config wrangler.<name>.jsonc

   Without this secret, the webhook endpoint is off (it answers 404).

2. In GitHub, open the repository settings, then Webhooks, and add a webhook:

   • Payload URL: https://<your-worker>/hooks/github
   • Content type: application/json
   • Secret: the same secret
   • Events: Pushes, Pull requests and Deployment statuses

What Checkend does with each event#

The repository name selects the projects: all projects with that repository. Other events (for example ping) are ignored. When GitHub sends the same delivery again (the same X-GitHub-Delivery ID, within 7 days), Checkend does nothing. See POST /hooks/github for the responses.

Suspect commits#

When Checkend knows the commits of your deploys, the issue brief can name a suspect commit: a commit of the release where the issue was first seen, which changed the file of an in-app frame of the issue. The frame where the error was raised counts most. When two commits score the same, the later commit wins. A changed file matches when its repository path is the end of the frame path: src/cart.ts matches /srv/app/src/cart.ts. With source maps, Checkend uses the original file.

The project needs its repository (see Set the repository of the project). Checkend gets the commit data in one of two ways:

• The GitHub webhook (push events): each pushed commit, with its message, author (the GitHub login), time and changed files (at most 300 files and 2,000 characters of message for each commit). No token is needed.
• An optional GITHUB_TOKEN (a Worker secret): a GitHub token that can read the contents of the repository. A fine-grained token with Contents: Read-only is enough. Then, for each new deploy, the cron gets the commits since the previous deploy to the same environment, and the changed files of up to 20 of them (the newest). It handles one deploy in each run (every 5 minutes). If the GitHub API fails, Checkend skips the deploy and logs it.

With GITHUB_TOKEN, the commits that Checkend gets also count as contained in the deploy. So a fix in one of them is marked as deployed, and CI does not need to send commits.

The brief shows the suspect commit in its table, for example | Suspect commit | a1b2c3d4e5f6 by grace: changed src/cart.ts (frame cartTotal) |. The commit message is text from the repository, so the brief shows it in an UNTRUSTED block.

3Send the release in the SDK#

Use the same release name in your Sentry SDK and in the deploy report:

Sentry.init({ dsn: "https://<key>@<your-worker>/<project-id>", release: "my-app@1.4.1" });

When a release contains the fix#

A release contains the fix when one of its deploys is in the same environment as a deploy with the fix, and at or after that deploy. So the later releases contain the fix too.

A revert or a rollback deploy after the fix also counts. When the error comes back after it, the issue becomes regressed, because the fix is not in production any more.

Verification#

A Cron Trigger runs every 5 minutes. A deployed fix becomes verified when, for the quiet period, the issue had no event that can come from code with the fix. Events of an older release (a release with a reported deploy that does not contain the fix) do not count. The quiet period starts at the deploy (or at the last event, when that is later). The quiet period is VERIFY_QUIET_HOURS hours (the default is 24; see Settings and health). The activity timeline gets issue.verify.

Checkend uses 1 Cron Trigger. The Workers Free plan allows 5 for each account.

Regression#

A new event of a fix_pending or verified issue, in a release that contains the fix, makes the issue regressed. The fix outcome becomes regressed, and the activity timeline gets issue.regress.

Events of older releases do not regress the issue: old servers can still run while a deploy rolls out. A regressed issue has a higher priority: the score is multiplied by 2.

Fix outcomes#

GET /v1/fix-outcomes lists the fixes that have an outcome (verified or regressed), with the hours from the link to verification, and from the first event of the issue to verification. A team can use it to see how often fixes work. It needs the admin scope. See GET /v1/fix-outcomes.

Edit this page on GitHubSource: guides/fix-loop.md