Auto-Healing

Auto-healing is Checksum’s batch repair system for tests that fail in CI and can’t be fixed by auto-recovery alone. When tests need code-level changes, Checksum creates an AI agent session that analyzes the failures, fixes the test code, and opens a PR with the healed tests.

How It Works

When healable test failures are detected, Checksum automatically creates an AI agent session to fix the tests. When healing completes, a PR with the fixes is opened to your repository for review.

---

Triage: Bug or Healable?

The first thing the healing agent does is triage every failure in the test run. For each failing test, the agent:

1. Reads the test results and error context (screenshots, error messages, stack traces)
2. Reads the relevant test code and application code
3. Classifies the failure as one of:
   • Test issue — the app is fine but the test needs fixing (selector drift, timing, stale setup, assertion drift). These proceed to the fix stage.
   • Application bug — a real defect in your product. The agent submits a bug verdict and tags the test with @bug in the source code. These are tracked in the Feature Health Dashboard for your team to address.

If triage finds no test-side issues to fix (every failure is an application bug, or there are no failures), the fix stage is skipped entirely — there’s nothing to heal.

Tests in the health dashboard can be in three states:

• Bug — confirmed application issue (won’t be healed)
• Clear — test is healthy
• Under Healing — a healing agent session is actively working on a fix

---

Triggering Healing

Choose your path: include autoHeal when you start a run (CLI --cksm-auto-heal, GitHub Action auto-heal: true, or API execution body) so healing starts when that run fails; call POST /auto-heal afterward if the run already finished without autoHeal; use the CLI flags when running npx checksumai test locally or in your own CI job.

Automatic (after a CI run)

When a CI run finishes with failures, Checksum can start healing automatically and open a PR with the fixes. Enable that for individual runs or for your whole app:

• Per-run — add --cksm-auto-heal to your npx checksumai test command in CI. When the run fails, healing starts and (by default) opens a PR. See Auto-healing failed CI runs and the test command flags.
• App-wide — Checksum can turn on automatic healing for your app so every failing CI run is healed without per-pipeline flags.

The repository, branch, and pull-request number are resolved automatically from the CI environment.

App-wide automatic healing is enabled by Checksum for your app — reach out to your Checksum representative to turn it on. Once enabled, set CHECKSUM_RUNTIME_REQUEST_REASON: cicd in your CI pipeline so those runs are recognized as CI runs and healed automatically.

Auto-heal when a run finishes (API)

The same per-run opt-in works when you dispatch runs via the public API instead of the CLI. Include an autoHeal block on any execution endpoint (/execution/suite, /execution/collection/:id, /execution/tests, or /public-api/v2/execution/grep):

{
  "autoHeal": {
    "autoCreatePR": true,
    "branch": "main",
    "prNumber": 42,
    "repoName": "acme-co/webapp",
    "metadata": { "source": "nightly" }
  }
}

Presence of autoHeal means “heal on failure.” Omit the block to opt out. When the run ends failed, Checksum starts a healing batch — no separate POST /auto-heal call needed.

Per-run autoHeal (CLI or API) takes priority over app-wide auto-heal-after-run settings for that run.

Auto-heal after scheduled runs

Checksum can auto-heal failures from scheduled test runs the same way as CI runs. That is a project-level setting — contact your Checksum team to enable it. Self-serve periodic scheduling in the product is not generally available yet.

From GitHub Actions

The Checksum AI Test Run action opts a run into auto-heal in a single step — auto-heal: true adds the heal block to the dispatch, and PR + repo are auto-resolved from the workflow context.

Via API

You can trigger healing programmatically for any test run:

curl -X POST https://api.checksum.ai/public-api/v1/auto-heal \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "testRunId": "<RUN_ID>",
    "autoCreatePR": true,
    "branch": "main",
    "prNumber": 42,
    "repoName": "<OWNER>/<REPO>"
  }'

Where healing runs

These parameters control what healing clones, where it commits, and where the resulting PR lands. They all act on the tests repository configured for your app.

Parameter	Type	Required	Description
testRunId	string	Yes	The test run to heal.
autoCreatePR	boolean	No	Open a PR in the tests repo when healing completes. Defaults to true.
branch	string	No	One branch for the tests repository — healing clones this branch in the tests repo, commits fixes there, and opens the healed PR against it. Defaults to the test run’s recorded branch when omitted; that default only works when the run was executed against a tests-repo branch. If the run was triggered from app-code CI (e.g. feature/foo), set branch to your tests repo’s integration branch (often main). There is no separate branch field per repository.

When healing completes and autoCreatePR is true, a PR is opened in the tests repo from a generated checksumai/<id> branch into branch.

Runs dispatched via API or the GitHub Action can also pass a top-level branch on grep execution — that controls which branch is checked out for the test run, not where the heal PR lands. Heal targeting always uses autoHeal.branch (or the heal API’s branch field).

Where to post status (correlation)

These parameters don’t affect cloning or where the healed PR lands — they only tell Checksum which repository hosts the PR that should receive healing progress comments.

Parameter	Type	Required	Description
prNumber	number	No	Existing PR to associate this healing batch with. Healing progress comments are posted on that PR. Does not change the clone target or the healed-PR target — both still use branch in the tests repo. Pair with repoName.
repoName	string	Conditional	<owner>/<repo> of the repo that contains prNumber (usually your app-code repo when CI ran from application code; can be the tests repo if the source PR is there). Required when autoCreatePR is true (the default) or when prNumber is set. Matched against git integrations connected to your app — not a second branch selector.
metadata	object	No	Arbitrary key-value pairs included as context in the agent’s prompt.

Because autoCreatePR defaults to true, repoName is required for nearly every call — only omit it if you explicitly pass "autoCreatePR": false and don’t pass prNumber.

Poll healing progress:

curl https://api.checksum.ai/public-api/v1/auto-heal/batch/<BATCH_ID> \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

See the API Reference for full details.

Deep vs Standard Healing

Like test generation, healing supports two modes:

Standard Healing

The agent triages failures, then fixes the healable ones:

Triage → Fix

• Triage — classifies each failure as an application bug or a test issue. Submits verdicts for real bugs.
• Fix — fixes test-local issues (selectors, timing, setup, assertions), runs the tests to verify, and commits the changes.

Best for: simple selector changes, timing fixes, minor assertion updates.

Deep Healing

Thorough analysis — the agent first plans the fixes, builds understanding of what changed in your app, then implements repairs.

Interview/Plan → Knowledge Base Update → Implementation → Review → Checksumify → Verify

Best for: major app refactors, flow changes, multiple related test failures.

---

What the Agent Fixes

Issue	How It’s Healed
Selector drift	Updates data-testid, role, or text selectors to match current DOM
Timing / waits	Replaces arbitrary waits with Playwright web-first assertions
Assertion mismatches	Updates expected values to match current app behavior
Setup / cleanup changes	Adjusts data setup and cleanup when API endpoints or data models change
Flow changes	Adds, removes, or reorders test steps to match the current user flow
Page layout changes	Adapts to restructured pages, moved elements, or new UI components
Authentication flow changes	Handles updated login, SSO, or multi-factor auth steps
API endpoint changes	Updates API calls in data setup/cleanup when backend endpoints change

---

Reviewing Healed Tests

When healing completes, you’ll receive a PR with the changes. Review it just like any other PR:

1. Read the diff — what was changed and why
2. Run the tests locally to verify they pass
3. Merge when satisfied

---

Healing Feedback

After reviewing healed tests, you can provide feedback on the healing quality. This helps Checksum improve its healing accuracy over time.

---

Notifications

Auto-heal batches emit two events that can be routed to your team’s chat tools via Notification Connectors:

• Auto-Heal Started — fires when a healing batch kicks off after a failing run.
• Auto-Heal Completed — fires when the batch finishes (every session reached a terminal state, or the batch failed).

Both are enabled by default for new connectors.

---

Next Steps

• View test health in the dashboard
• Run tests in CI
• Route auto-heal events to Slack, Teams, Discord, or Google Chat