Auto-Healing

Auto-healing is Checksum’s batch repair system for tests that fail in CI and can’t be fixed by auto-recovery alone. When tests need code-level changes, Checksum creates an AI agent session that analyzes the failures, fixes the test code, and opens a PR with the healed tests.

How It Works

When healable test failures are detected, Checksum automatically creates an AI agent session to fix the tests. When healing completes, a PR with the fixes is opened to your repository for review.

---

Triage: Bug or Healable?

The first thing the healing agent does is triage every failure in the test run. For each failing test, the agent:

1. Reads the test results and error context (screenshots, error messages, stack traces)
2. Reads the relevant test code and application code
3. Classifies the failure as one of:
   • Test issue — the app is fine but the test needs fixing (selector drift, timing, stale setup, assertion drift). These proceed to the fix stage.
   • Application bug — a real defect in your product. The agent submits a bug verdict and tags the test with @bug in the source code. These are tracked in the Feature Health Dashboard for your team to address.

If every failure in the run is classified as an application bug, the fix stage is skipped entirely — there’s nothing to heal.

Tests in the health dashboard can be in three states:

• Bug — confirmed application issue (won’t be healed)
• Clear — test is healthy
• Under Healing — a healing agent session is actively working on a fix

---

Triggering Healing

Automatic

Healing triggers automatically when tests fail in CI and the failures are classified as healable.

Via API

You can trigger healing programmatically for any test run:

curl -X POST https://api.checksum.ai/public-api/v1/auto-heal \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "testRunId": "<RUN_ID>",
    "autoCreatePR": true,
    "branch": "main",
    "prNumber": 42
  }'

Parameter	Type	Required	Description
testRunId	string	Yes	The test run to heal
autoCreatePR	boolean	No	Auto-create a PR with fixes (default: true)
branch	string	No	Branch to create the PR against
prNumber	number	No	Associate healing with an existing PR
metadata	object	No	Arbitrary key-value pairs included as context in the agent’s prompt

When healing completes, a PR with the fixes is automatically opened to your repository.

Poll healing progress:

curl https://api.checksum.ai/public-api/v1/auto-heal/batch/<BATCH_ID> \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

See the API Reference for full details.

Deep vs Standard Healing

Like test generation, healing supports two modes:

Standard Healing

The agent triages failures, then fixes the healable ones:

Triage → Fix

• Triage — classifies each failure as an application bug or a test issue. Submits verdicts for real bugs.
• Fix — fixes test-local issues (selectors, timing, setup, assertions), runs the tests to verify, and commits the changes.

Best for: simple selector changes, timing fixes, minor assertion updates.

Deep Healing

Thorough analysis — the agent first plans the fixes, builds understanding of what changed in your app, then implements repairs.

Interview/Plan → Knowledge Base Update → Implementation → Review → Checksumify → Verify

Best for: major app refactors, flow changes, multiple related test failures.

---

What the Agent Fixes

Issue	How It’s Healed
Selector drift	Updates data-testid, role, or text selectors to match current DOM
Timing / waits	Replaces arbitrary waits with Playwright web-first assertions
Assertion mismatches	Updates expected values to match current app behavior
Setup / cleanup changes	Adjusts data setup and cleanup when API endpoints or data models change
Flow changes	Adds, removes, or reorders test steps to match the current user flow
Page layout changes	Adapts to restructured pages, moved elements, or new UI components
Authentication flow changes	Handles updated login, SSO, or multi-factor auth steps
API endpoint changes	Updates API calls in data setup/cleanup when backend endpoints change

---

Reviewing Healed Tests

When healing completes, you’ll receive a PR with the changes. Review it just like any other PR:

1. Read the diff — what was changed and why
2. Run the tests locally to verify they pass
3. Merge when satisfied

---

Healing Feedback

After reviewing healed tests, you can provide feedback on the healing quality. This helps Checksum improve its healing accuracy over time.

---

Next Steps

• View test health in the dashboard
• Run tests in CI