AI can draft the diagnosis. It must not rewrite the evidence.

The useful outcome is control, not a faster guess. AI can compress logs, normalize a failure bundle, and draft competing explanations. It should not be allowed to turn those drafts into a root cause, a code change, or release authority.

Consider a login test that times out in CI and passes on a laptop. The visible symptom is compatible with several causes: an authentication dependency, a stale selector, a fixture race, an environment difference, or a product regression. A fluent explanation can sound decisive while proving none of them.

The risk grows when one automation step can read the failure, edit the test, rerun it, and report success. The original signal can disappear inside the repair. A green rerun then tells the reviewer only that the modified system passed once. It does not establish why the original run failed or whether the protected requirement still holds.

Use an evidence boundary

An evidence boundary is a small structural rule: each kind of statement gets its own field, state, and owner. The model may help prepare the fields, but it cannot silently move information between them.

This separation changes the question from “What does the model think happened?” to “What is established, what remains open, and what would discriminate between the live explanations?” That is a question a reviewer can inspect.

A copyable decision record

The following template is deliberately incomplete until evidence exists. Empty fields are safer than invented certainty. Store references to controlled artifacts rather than copying secrets, personal data, customer content, or unrestricted logs into the record.

decision_record:
  state: unresolved

  run:
    test_id: <stable test identifier>
    test_version: <version or commit ref>
    deployment_ref: <deployed version>
    environment_ref: <environment snapshot>
    expected_contract_ref: <requirement or acceptance rule>

  observed_evidence:
    - claim: <fact established by an artifact>
      evidence_ref: <log, trace, screenshot, or report ref>

  hypotheses:
    - id: H1
      statement: <testable explanation, not a cause claim>
      supporting_refs: []
      contradicting_refs: []
      missing_evidence: []
      falsification_step: <smallest discriminating check>

  missing_evidence:
    - item: <what is needed>
      owner: <role responsible for collection>

  proposal:
    diff_ref: null
    protected_behavior_ref: <what must not be weakened>
    assumptions: []
    rollback_ref: null

  validation:
    isolated_check_ref: null
    negative_control_ref: null
    related_regression_ref: null
    original_failure_retained: true

  authority:
    diagnostic_owner: <role>
    change_reviewer: <role>
    release_authority: <role outside the runner>

The record does not need to be YAML. The important property is that a system cannot satisfy a missing evidence field by producing more prose. A reference must resolve to an authorized artifact, a hypothesis must remain falsifiable, and an approval must come from the configured role.

Worked example: a login timeout

Start by refusing the repair. The current record can establish only what the preserved run artifacts show. An honest first pass might look like this:

That answer is less dramatic than a generated root cause. It is also more useful. It preserves the failed observation, makes the environment difference explicit, and routes the next action toward evidence instead of toward a convenient green check.

Add structural gates, not confidence scores

A model confidence score does not establish causality. Use gates that inspect record structure and referenced evidence. A small guard can prevent an automation path from opening a change before the required states exist:

def eligible_for_change_review(record: dict) -> bool:
    return all(
        [
            record["state"] == "cause_reviewed",
            bool(record["proposal"]["diff_ref"]),
            bool(record["proposal"]["protected_behavior_ref"]),
            bool(record["validation"]["isolated_check_ref"]),
            bool(record["validation"]["negative_control_ref"]),
            record["validation"]["original_failure_retained"] is True,
            bool(record["authority"]["change_reviewer"]),
        ]
    )

This function does not decide that the change is correct. It only rejects a structurally incomplete transition. The cited artifacts still need to be real, authorized, and reviewed. The check belongs beside schema validation, reference resolution, permission checks, and normal code review.

Three implementation details that matter

Resolve references against an authorized manifest

An artifact reference is useful only if the reviewer can resolve it to the intended version. Keep a manifest that binds each reference to a content hash, collection time, source system, access class, and retention rule. Reject unknown references. Do not let a generated response introduce a new log location or quietly point at a later rerun.

Make invalid state transitions impossible

Treat unresolved, hypotheses_ready, cause_reviewed, and change_pending as different states, not labels in prose. A transition should name the actor, evidence references, reason, and policy version. If the record is unresolved, another confident paragraph cannot move it to a reviewed cause.

Keep evidence reads separate from production writes

Collection and drafting can often remain read-only. Ticket creation, code changes, retries against shared environments, rollback, and release are writes with different permissions and failure modes. Put them behind separate adapters and explicit approval. Give every allowed write an idempotency key and retain the external record identifier, so a network retry cannot create a duplicate issue or repeat an operational action.

These controls sound mundane because they are. That is their advantage. They turn reviewability into a system property instead of a request in the prompt.

Keep collection, diagnosis, and action separate

1. Freeze the run context. Record the test, commit, deployment, environment, dependency, and expected contract before retries or edits.
2. Collect approved evidence. Preserve the original failure and list what is missing. Redact or reference protected data according to policy.
3. Draft competing hypotheses. Require supporting and contradicting references plus one discriminating check for each candidate.
4. Run the narrowest authorized check. State in advance what result would support, reject, or leave each hypothesis unresolved.
5. Review the cause separately. A plausible explanation, temporal coincidence, or similar old incident is not a completed root cause.
6. Draft a bounded change. Preserve protected assertions, identify assumptions, add negative controls, and keep rollback explicit.
7. Validate outside the proposal step. Retain the original failure, test intended positive behavior, and prove that the negative control still fails.
8. Route the release decision. The runner can publish evidence. A named authority decides whether the change moves forward.

A short review checklist

• Can every observed fact be traced to a versioned artifact?
• Are facts visually and structurally separate from hypotheses?
• Does every hypothesis have a check that could prove it wrong?
• Is missing evidence allowed to keep the record unresolved?
• Can a proposed repair weaken an assertion, add a blind retry, or replace the original failure?
• Are validation evidence and the original failure retained together?
• Are diagnosis, change approval, and release authority assigned to explicit roles?

A good AI-assisted testing workflow does not eliminate uncertainty. It prevents uncertainty from being renamed as certainty while the team is under pressure. The result may take longer than an instant guess, but it leaves a trace that a reviewer can inspect instead of asking the team to defend an untraceable decision later.

Continue the method

Build the full path from failure triage to team-scale review.

The Complete AI-Native QA Series connects evidence-first triage, AI application testing, bounded automation, validation, and operating controls across L1 START through L4 SCALE.

See the complete AI-Native QA Series →