AI workflow QA checklist: test agents before production.

Short version

An AI workflow QA checklist is the operating test plan for a production AI workflow. It defines the source records to test, the expected outputs, the tool calls that must be valid, the approvals that must be reviewable, the exceptions that must route cleanly, and the launch gates that decide whether the workflow can act.

This matters when AI touches sales qualification, CRM updates, support queues, document intake, marketing operations, inbox triage, meeting follow-up, internal reporting, or founder operations. The model can sound right while the workflow still writes the wrong field, misses the account context, skips a required approval, or hides the evidence an owner needs to trust the result.

If you are still choosing the first workflow, start with the AI workflow audit checklist. If the workflow is already selected, pair this guide with the AI workflow rollout plan, AI workflow telemetry, and AI workflow permissions audit.

Production QA is where AI ambition becomes an operating system instead of a clever prototype.

Why AI workflow QA breaks

Teams usually test the happy path because it is the path that made the automation exciting. A lead arrives with all fields filled. The support ticket has a clean category. The meeting transcript names every owner. The document is formatted correctly. The tool call succeeds. The reviewer approves.

Production rarely behaves like that. The first serious QA pass should hunt for operating failure, not just model weakness.

• Thin fixtures: test records only cover neat examples, so the workflow never sees duplicates, missing context, stale data, attachments, contradictions, or ambiguous ownership.
• Unclear expected output: nobody has written what a good classification, draft, CRM note, escalation packet, or tool payload should look like.
• Prompt-only testing: the team checks model text but not retrieval, connector permissions, field validation, idempotency, write-back behavior, or reviewer edits.
• No owner review loop: subject-matter owners approve the concept but never inspect enough real outputs to define acceptable quality.
• Weak exception tests: the workflow has no proof that it can pause, route, retry, escalate, or recover when source data is wrong or a tool call fails.
• Launch criteria by optimism: the team moves to production because the demo feels useful, not because the workflow passed a stated gate.

The fix is not a huge enterprise QA process. The fix is a small, explicit test harness around the workflow you actually want to ship.

Build test cases from real operating work

Start with 20 to 50 representative cases from the workflow. Include the normal work, the awkward work, and the work you do not want an agent to handle automatically. A sales workflow needs good-fit leads, bad-fit leads, duplicates, missing firmographics, unclear buying signals, stale CRM data, and conflicting transcript notes. A support workflow needs easy tickets, angry customers, policy-risk cases, account-context gaps, refund asks, and escalation-worthy failures.

Each fixture should include the input, source records, expected output, allowed action, approval rule, and reason. Without that, the team is grading vibes instead of workflow behavior.

Test the workflow in layers

A production workflow is a chain. Testing only the final answer hides which link is weak. Split QA into layers so owners and engineers can identify whether the problem is input quality, retrieval, prompt behavior, tool-call construction, approval review, or operating policy.

Automated evals are useful for repeatable checks: classification accuracy, required fields, schema validity, citation coverage, policy labels, tone constraints, duplicate detection, and regression drift. Human review is still needed for business judgment, account nuance, customer trust, and irreversible actions.

QA the tool calls, not just the model output

The most expensive failure often happens after the text looks good. The workflow writes the wrong CRM field, creates duplicate tasks, sends the wrong email variant, updates a stale ticket, overwrites a human note, or retries a non-idempotent action.

• Dry-run first: capture the exact payload the workflow would send before allowing approved write-back.
• Validate field mapping: confirm every field, enum, owner id, timestamp, source link, and external-system object id is correct.
• Check idempotency: repeated runs should not create duplicate contacts, tasks, notes, tickets, invoices, documents, or outbound messages.
• Separate draft from write: drafts can be useful even when writes must wait for review, permissions, or better telemetry.
• Force visible failures: permission errors, stale records, failed writes, and rejected payloads should create exceptions instead of silent partial state.
• Record owner edits: reviewer changes are training signal for prompts, evals, policy rules, and launch readiness.

For permission boundaries, use the AI workflow permissions audit. For review queues, use AI approval workflow automation. For failures, use AI workflow exception handling.

Where stack fit belongs in QA

QA exposes whether the existing stack can support the workflow or whether the team needs custom infrastructure. Some tools make dry-run payloads, scoped tokens, approval queues, audit logs, field validation, and rollback easy. Others hide too much behavior behind broad credentials, weak APIs, or manual workarounds.

Use Purple Orange Stack's AI automation audit page as supporting context when QA needs to evaluate process standardization, tool integration, measurement, team capability, prioritization, and stack readiness. That is relevant here because a QA plan must prove whether the workflow can be safely implemented inside the current CRM, helpdesk, marketing ops, sales, docs, or project-management stack.

A first system can often stay inside the current stack if the workflow only needs read-only context, reviewed drafts, and simple routing. Custom infrastructure becomes the stronger choice when multiple workflows need shared fixtures, evals, connectors, approval gates, telemetry, incident logs, and engineering handoff. That is the boundary between a narrow AI automation sprint and a broader AI operations buildout or production AI infrastructure.

Define launch gates before the pilot starts

The team should know what must be true before the workflow moves from shadow mode to draft-first review, from draft-first review to approved write-back, and from approved write-back to narrow autonomy.

• Fixture pass rate: the workflow passes the agreed normal, edge, missing-context, contradiction, and do-not-automate cases.
• Owner edit rate: reviewers are not rewriting most outputs, correcting core facts, or changing the same field repeatedly.
• Exception quality: low-confidence, blocked, risky, or failed-write cases route to the right owner with enough evidence.
• Tool-call validity: dry-run payloads are valid, scoped, idempotent, and traceable before write-back.
• Rollback readiness: the team can pause the workflow, revert a write, disable a connector, and notify owners.
• Telemetry visibility: runs, approvals, exceptions, owner edits, latency, cost, and outcome quality are visible after launch.

These gates should be boring and explicit. If the team cannot say what evidence unlocks the next stage, the workflow is not ready for more autonomy.

What the QA checklist should produce

The output should be a launch artifact the founder, ops owner, and builder can use. It should make the go/no-go decision visible without turning the project into a compliance ceremony.

This is a strong candidate for a free Purple Orange AI workflow audit. Bring one workflow that looks close to production but still has uncertainty around source data, tool calls, approvals, exceptions, or launch readiness. The audit should return the QA plan, risk rating, fixture gaps, telemetry needs, and a yes/no on whether the next move is cleanup, sprint, operations buildout, or production AI infrastructure.