CI Integration - Checksum AI

Overview

The typical CI setup:

Get your API key from the Checksum web app and store it as a CI secret
Choose a running strategy (on PR merge, on schedule, manual trigger)
If tests are in a separate repo, create a Personal Access Token (PAT) with repo access
Add the CI configuration below

GitHub Actions

Basic Setup

Create .github/workflows/checksum-tests.yml:

name: Run Checksum Tests

on:
  # Choose your triggers:
  # schedule:
  #   - cron: '0 0 * * *'  # Runs every day at 00:00 UTC
  workflow_dispatch:        # Allows manual triggering

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Install npm dependencies
        run: npm install

      - name: Install Playwright with dependencies
        run: npx playwright install --with-deps

      - name: Download .env from Checksum
        run: npx checksumai dotenv --download --api-key="${{ secrets.CHECKSUM_API_KEY }}"

      - name: Run Checksum tests
        run: npx checksumai test
        env:
          CHECKSUM_API_KEY: ${{ secrets.CHECKSUM_API_KEY }}
          USERNAME: ${{ secrets.USERNAME }}
          PASSWORD: ${{ secrets.PASSWORD }}
          LOGIN_URL: ${{ secrets.LOGIN_URL }}
          BASE_URL: ${{ secrets.BASE_URL }}
          CI: true

Cross-Repo Setup

If your tests live in a different repository than the one running the action, checkout the test repo using a Personal Access Token:

steps:
  - uses: actions/checkout@v4

  - name: Checkout test repo
    uses: actions/checkout@v4
    with:
      repository: org/your-test-repo
      ref: main
      token: ${{ secrets.TEST_REPO_PAT }}
      path: .

  # ... rest of the steps

Setting Up Secrets

Go to your GitHub repository → Settings → Secrets and variables → Actions
Add the following secrets:
- CHECKSUM_API_KEY — your Checksum API key
- USERNAME — test user credentials
- PASSWORD — test user password
- BASE_URL — your application URL
- LOGIN_URL — your login page URL
- TEST_REPO_PAT (if cross-repo) — Personal Access Token with repo permissions

Testing the Action

Commit the workflow file to your main branch
Go to Actions → select your workflow → Run Workflow
Monitor the run to confirm tests execute successfully

GitHub Action (no CLI install)

If you don’t need to run tests on your own runner — e.g., your CI just needs to trigger a Checksum run and let Checksum execute it — use checksum-ai/test-run-action. It wraps the public-API execution endpoints in a single step, with optional auto-heal-on-failure built in.

Quick start

name: Checksum tests
on: pull_request

permissions:
  contents: read
  pull-requests: read

jobs:
  checksum:
    runs-on: ubuntu-latest
    steps:
      - uses: checksum-ai/test-run-action@v1
        with:
          api-key: ${{ secrets.CHECKSUM_API_KEY }}
          grep: 'checkout'
          auto-heal: true

That’s the whole step. The action auto-detects the source PR (from the event payload on pull_request, or via the GitHub API on push) and threads it through to auto-heal so progress comments land on the right PR.

Execution modes

Set exactly one of the following inputs to pick which tests to run:

Input	Endpoint	Use when
`grep`	`POST /public-api/v2/execution/grep`	You want to filter tests by name pattern (substring or regex).
`suite-ids`	`POST /public-api/v1/execution/suite`	You want to run one or more specific test suites. Comma-separated UUIDs.
`test-ids`	`POST /public-api/v1/execution/tests`	You want to run a specific list of tests by UUID.
`collection-id`	`POST /public-api/v1/execution/collection/:id`	You want to run a saved collection.

Auto-heal on failure

Set auto-heal: true to run Auto-Healing when the test run fails. Healing sessions start automatically and (by default) push fixes as a PR. Healing progress is reported as a comment on the originating PR — no extra wiring needed when running on pull_request events. To dispatch heal sessions without auto-creating a PR:

- uses: checksum-ai/test-run-action@v1
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    auto-create-pr: false

Wait for completion (gate the workflow)

By default the action exits as soon as the dispatch is accepted (~15s) — you get notified about results via the standard PR comment. Set wait: true if you want the workflow check itself to gate on the test outcome:

- uses: checksum-ai/test-run-action@v1
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    wait: true

Final status	Step exit
`passed`, `healed`	success
`failed`, `process-error`, `cancelled`	failure
timeout	failure

wait: true keeps a runner allocated for the full test-run duration (typically 5–25 minutes). When runner-minute cost matters, prefer wait: false and use the standard PR-comment notification.

Per-PR preview environments

Pass env-overrides (grep mode only) to inject per-run environment variables — handy when each PR is deployed to its own preview URL:

- uses: checksum-ai/test-run-action@v1
  with:
    api-key: ${{ secrets.CHECKSUM_API_KEY }}
    grep: 'checkout'
    auto-heal: true
    env-overrides: |
      {"BASE_URL": "https://pr-${{ github.event.pull_request.number }}.preview.example.com"}

Full reference

The complete inputs/outputs reference and changelog live in the action’s README. Pin to a major-version tag (@v1) for safety, or to a specific release (@v1.0.0) if you want a frozen reference.

GitLab CI/CD

Basic Setup

Add to your .gitlab-ci.yml:

image: node:20-bookworm

stages:
  - test

run-checksum-tests:
  stage: test
  rules:
    # - if: '$CI_PIPELINE_SOURCE == "schedule"'  # Runs for scheduled pipelines
    - if: '$CI_PIPELINE_SOURCE == "web"'          # Allows manual triggering
      when: manual

  before_script:
    - npm ci
    - npx playwright install --with-deps
    - npx checksumai dotenv --download --api-key="${CHECKSUM_API_KEY}"

  script:
    - npx checksumai test

  variables:
    CHECKSUM_API_KEY: $CHECKSUM_API_KEY
    USERNAME: $USERNAME
    PASSWORD: $PASSWORD
    LOGIN_URL: $LOGIN_URL
    BASE_URL: $BASE_URL
    CI: "true"

  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/

Cross-Repo Setup

If tests are in a separate repository:

  before_script:
    - git clone "https://gitlab-ci-token:${TEST_REPO_PAT}@gitlab.com/org/your-test-repo.git" tests
    - cd tests
    - npm ci
    - npx playwright install --with-deps
    - npx checksumai dotenv --download --api-key="${CHECKSUM_API_KEY}"

Setting Up CI/CD Variables

Go to your GitLab project → Settings → CI/CD → Variables
Add the same variables as listed in the GitHub section above

Programmatic CI Integration (API)

Instead of running the CLI directly, you can trigger and manage test runs programmatically using the Checksum API. This is useful for custom CI pipelines, orchestration tools, or when you need to chain test runs with other operations.

If you’re on GitHub Actions, the Checksum AI Test Run action wraps these endpoints in a single step — including the auto-heal trigger — and is usually a cleaner starting point than the raw API.

Run Tests

# Run all tests
curl -X POST https://api.checksum.ai/public-api/v1/execution/suite \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"
# Response: { "name": "job-name-12345" }

Poll for Results

# Check status
curl https://api.checksum.ai/public-api/v1/execution/status/job-name-12345 \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"
# Response: { "status": "passed", "passed": 45, "failed": 2, "recovered": 1, "bug": 0 }

Get Detailed Results

curl https://api.checksum.ai/public-api/v1/test-runs/<RUN_ID>/results \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

Trigger Auto-Healing

curl -X POST https://api.checksum.ai/public-api/v1/auto-heal \
  -H "Authorization: Bearer $CHECKSUM_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "testRunId": "<RUN_ID>",
    "autoCreatePR": true,
    "branch": "main",
    "prNumber": 42,
    "repoName": "<OWNER>/<REPO>"
  }'
# Response: { "batchId": "...", "sessionIds": [...], "failureCount": 3, "testIds": [...] }

Poll Healing Progress

curl https://api.checksum.ai/public-api/v1/auto-heal/batch/<BATCH_ID> \
  -H "Authorization: Bearer $CHECKSUM_API_KEY"

For the complete API documentation, see API Reference.

Auto-healing failed CI runs

Checksum can fix failing tests after a CI run and open a PR with the fixes. Enable auto-heal per run or for your whole app:

Per-run, from the test command

Add the auto-heal flag to your test command. This works the same on GitHub Actions and GitLab CI — the repository, branch, and pull-request number are detected automatically from the CI environment:

- name: Run Checksum tests
  run: npx checksumai test --cksm-auto-heal
  env:
    CHECKSUM_API_KEY: ${{ secrets.CHECKSUM_API_KEY }}
    BASE_URL: ${{ secrets.BASE_URL }}
    CI: true

When the run finishes with failures, healing starts automatically and (by default) opens a PR with the fixes. See the CLI Reference for the full --cksm-auto-heal* flag set, and Auto-Healing for how healing works.

App-wide

Checksum can enable automatic healing for your whole app, so every failing CI run is healed without adding flags to each pipeline. Contact your Checksum representative to turn this on for your app.

When app-wide healing is enabled, tell Checksum which runs to treat as CI runs by setting this environment variable in your pipeline:

env:
  CHECKSUM_RUNTIME_REQUEST_REASON: cicd

With that set, every failing run from this pipeline is healed automatically — no per-command flags needed.

Running Strategies

Strategy	When to Use	Configuration
Manual trigger	Testing and debugging	`workflow_dispatch` (GitHub) / `when: manual` (GitLab)
On schedule	Regular health checks	`schedule: cron` (GitHub) / `$CI_PIPELINE_SOURCE == "schedule"` (GitLab)
On PR merge	Verify after deployments	`push: branches: [main]` (GitHub) / `$CI_PIPELINE_SOURCE == "push"` (GitLab)

Start with manual triggers to verify your setup, then add scheduled or merge-triggered runs.

​Overview

​GitHub Actions

​Basic Setup

​Cross-Repo Setup

​Setting Up Secrets

​Testing the Action

​GitHub Action (no CLI install)

​Quick start

​Execution modes

​Auto-heal on failure

​Wait for completion (gate the workflow)

​Per-PR preview environments

​Full reference

​GitLab CI/CD

​Basic Setup

​Cross-Repo Setup

​Setting Up CI/CD Variables

​Programmatic CI Integration (API)

​Run Tests

​Poll for Results

​Get Detailed Results

​Trigger Auto-Healing

​Poll Healing Progress

​Auto-healing failed CI runs

​Per-run, from the test command

​App-wide

​Running Strategies

Overview

GitHub Actions

Basic Setup

Cross-Repo Setup

Setting Up Secrets

Testing the Action

GitHub Action (no CLI install)

Quick start

Execution modes

Auto-heal on failure

Wait for completion (gate the workflow)

Per-PR preview environments

Full reference

GitLab CI/CD

Basic Setup

Cross-Repo Setup

Setting Up CI/CD Variables

Programmatic CI Integration (API)

Run Tests

Poll for Results

Get Detailed Results

Trigger Auto-Healing

Poll Healing Progress

Auto-healing failed CI runs

Per-run, from the test command

App-wide

Running Strategies