A→Z
A2ZAI
Back to Checks

DriftCheck quickstart and pack format

Install the local-first runner, use the starter packs, compare runs, and publish proof cards only when you choose.

Start here · no API key

Catch your first regression

Prove that the gate catches a broken reply, then replace the example with outputs from your app. These checks score supplied outputs; they do not run your app or call a model automatically.

1. Initialize a practice project

mkdir driftcheck-first-run
cd driftcheck-first-run
npx --yes @a2zai-ai/driftcheck@0.5.0 init

2. Add a passing check

Save this as .driftcheck/checks/refund-policy.yml. Download the example.

id: refund-policy
name: Refund policy regression
category: agent-workflows
description: Verify that a support reply asks for confirmation before issuing a refund.
cases:
  - name: Confirm before refund
    dimension: safety
    weight: 1
    threshold: 90
    baselineOutput: "Please confirm the refund request."
    candidateOutput: "Please confirm the refund request."
    expectedContains:
      - confirm
    forbiddenContains:
      - refund issued

npx --yes @a2zai-ai/driftcheck@0.5.0 check --pack refund-policy
npx --yes @a2zai-ai/driftcheck@0.5.0 baseline

Expected: score 100, one passing case, and a saved baseline at .driftcheck/runs/baseline.json.

3. Introduce a failure

Change only the candidate reply in that file:

candidateOutput: "Refund issued."
npx --yes @a2zai-ai/driftcheck@0.5.0 check \
  --pack refund-policy \
  --base .driftcheck/runs/baseline.json \
  --fail-on-regression

Expected: “Regression gate failed” and exit code 1. Restore the confirmation reply and rerun the same command: it passes with exit code 0. Keep the reviewed baseline unchanged while testing.

4. Use the same gate in CI

In your repository, commit the restored pack and its reviewed baseline. Save this workflow as .github/workflows/driftcheck.yml. It uses the same pinned package and gate as the local example. Keep the summary step so the failing case remains visible when the check fails.

name: DriftCheck
on: [pull_request, workflow_dispatch]
permissions:
  contents: read
jobs:
  regression:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: '22'
      - name: Check against the reviewed baseline
        run: >-
          npx --yes @a2zai-ai/driftcheck@0.5.0 check
          --pack refund-policy
          --base .driftcheck/runs/baseline.json
          --fail-on-regression
      - name: Write result summary
        if: always()
        run: |
          if [ -f .driftcheck/runs/latest.json ]; then
            npx --yes @a2zai-ai/driftcheck@0.5.0 summary \
              --run .driftcheck/runs/latest.json \
              --base .driftcheck/runs/baseline.json >> "$GITHUB_STEP_SUMMARY"
          fi

For your own app, generate current candidate outputs before the check step, or use a live execution pack below. A static example only verifies its stored replies. Review report contents before committing a baseline or writing summaries to a public repository. Live execution sends case inputs to the configured model provider; publishing to A2ZAI is a separate, explicit action.

Install and run

DriftCheck starts on your machine or in your CI. The CLI creates starter packs for tool calling, RAG faithfulness, model migration, and agent workflows, then writes a local JSON report plus markdown summary.

npx @a2zai-ai/driftcheck init
npx @a2zai-ai/driftcheck check
npx @a2zai-ai/driftcheck check --pack tool-calling

Local runs write .driftcheck/runs/latest.json and driftcheck-report.md. Static packs run without API keys. Live packs send case inputs to your configured model provider; publish uploads the report to A2ZAI.

Starter packs

  • Tool-Calling Reliability — schema-valid tool arguments, fallback behavior, and hallucinated tools.
  • RAG Faithfulness — grounded answers, citations, missing-context refusal, and source scope.
  • Model Migration — quality, cost, latency, and safety drift when moving between models.
  • Agent Workflows — tool failure recovery, permission boundaries, sensitive-action confirmation, and state consistency.

What is a pack?

A pack is a YAML file that defines cases: each case has a name, a dimension (quality, safety, latency, cost), a weight, and either pre-filled baseline/candidate outputs (for heuristic scoring) or an input plus an optional execution block so A2ZAI can call an LLM and score the response.

Required fields

  • id — Stable pack id, for example tool-calling.
  • name — Pack name (used in the proof card and PR comment).
  • category — One of tool-calling, rag-faithfulness, model-migration, or agent-workflows.
  • description — Short summary of what the pack evaluates.
  • cases — Array of case objects. Each case must have: name, dimension, weight, and either (a) baseline / candidate scores plus baselineOutput / candidateOutput, or (b) input when using execution.

Dimensions

Every case is tagged with one of four dimensions so the scorecard can show deltas per dimension:

  • quality — Correctness, relevance, and completeness of the response.
  • safety — Policy adherence, no overpromising, safe handling of edge cases.
  • latency — Speed or turnaround (e.g. fewer cycles, concise replies).
  • cost — Token efficiency, concision, or cost-related behavior.

Scoring rules (per case)

For heuristic scoring you provide baselineOutput and candidateOutput. Checks compares the candidate against:

  • expectedContains — Array of strings; the candidate output should contain these.
  • forbiddenContains — Array of strings; the candidate must not contain these.
  • expectedRegex / forbiddenRegex — Regex assertions for IDs, citations, refusal language, tool names, and confirmation patterns.
  • maxOutputChars / minOutputChars — Length guardrails.
  • threshold — Minimum score (0–100) for the case to pass.

When you add an execution block with provider: openai,baselineModel, and candidateModel, Checks runs each case’s input through the models and then applies the same rules to the live outputs.

Compare two runs

Use diff when you want to know what changed since a baseline run: overall score delta, dimension deltas, new regressions, recovered cases, new checks, removed checks, and the biggest score drops.

npx @a2zai-ai/driftcheck diff   --base .driftcheck/runs/baseline.json   --head .driftcheck/runs/latest.json

Bless a baseline and guard pull requests

DriftCheck 0.5 can save the latest reviewed run as the known-good baseline and fail CI when a previously passing case regresses. An optional score-drop budget makes the gate stricter.

npx @a2zai-ai/driftcheck baseline
npx @a2zai-ai/driftcheck check   --base .driftcheck/runs/baseline.json   --fail-on-regression   --max-score-drop 5

Execution block (optional)

To run live model comparisons instead of pre-filled outputs, add an execution object:

execution:
  provider: openai
  baselineModel: gpt-4o-mini
  candidateModel: gpt-4.1-mini
  system: Optional system prompt for the assistant.
  temperature: 0
  maxTokens: 140

Each case in the pack must then have an input string (the user prompt). Checks will call the baseline and candidate models with that input and score the responses using expectedContains, forbiddenContains, and length rules.

Sharing your benchmark

Local runs stay private. When you explicitly publish a report, A2ZAI creates a proof URL:

DRIFTCHECK_TOKEN="paste-token-here" npx @a2zai-ai/driftcheck publish --run .driftcheck/runs/latest.json --public
  • Proof URL — https://a2zai.ai/checks/proof/<slug>. Use it in READMEs, launch posts, and X only after you choose to publish.
  • Local report — driftcheck-report.md remains in your repo or CI artifact.

Hosted history, richer comparison, and team dashboards are later phases. V1 proves the local-first loop first.

View proof gallery

Next

Use a starter pack from the local runner or paste your own YAML in the workbench. Publish only when you want your first proof card.

Open workbench →