The 30-case test your AI pilot should pass

A pilot can look convincing while doing the wrong work. A booking assistant might sound helpful, repeat the requested time, and still put the appointment on the wrong calendar. Before expanding a pilot, give the team a small set of cases they can run again after every change.

Thirty cases is a useful starting scope for a workshop, not a statistical guarantee. The goal is to turn "this seems good" into an explicit agreement about what the system must do.

Start with the record that should exist

Write the expected outcome before writing an ideal response. For a scheduling workflow, that could mean one appointment on the correct calendar, the correct contact attached, and no confirmation sent until the booking succeeds.

This distinction has a research precedent. The June 2024 tau-bench paper evaluates agents by comparing the final database state with the intended state. It also tests consistency across repeated attempts. A convincing conversation alone does not establish that the task was completed.

For your own pilot, identify the system of record and the few fields that determine success. A human reviewer should be able to inspect them without interpreting the model's explanation.

Build a small, uneven case set

Here is an illustrative starting set for a booking assistant. Change the counts to fit the operation.

Cases	Situation	What to check
12	Ordinary requests with available appointments	Correct contact, service, time zone, and calendar
6	Missing or ambiguous information	Clarification before a write
5	Requests the business cannot fulfill	Accurate explanation and an appropriate next step
4	Calendar failures, slow replies, or duplicate events	No duplicate booking or false confirmation
3	Requests for someone else's information	No unauthorized disclosure or action

Include the awkward details your team actually sees. Two contacts share a name. A caller changes the date after confirming it. A customer asks for "next Friday" near midnight in another time zone. An upstream service returns a timeout after accepting a request.

Use permissioned, minimized examples from the operation, or build synthetic cases that preserve the difficulty without copying personal data. Do not turn the test folder into an unprotected customer archive.

Separate hard failures from quality issues

An unauthorized data disclosure and an overly wordy greeting should not cancel each other out in an average score.

Mark a small set of failures as release blockers. Examples include writing to the wrong customer, making an unsupported commitment, or confirming an action that did not happen. Score conversational quality separately, with examples of an acceptable answer.

Anthropic's January 2026 guide to agent evaluations distinguishes tasks, repeated trials, graders, and final outcomes. It also describes combining code-based checks with model and human assessment. For this pilot, that suggests a practical division: inspect database changes automatically and let staff judge whether the explanation is clear.

Repeat the cases that can fail expensively

Run sensitive cases several times. Record each result instead of keeping the best attempt. A case that passes four times and creates a duplicate on the fifth is still unresolved.

Keep a separate holdout set that the developer does not continuously tune against. Add new production failures to the regression set, but keep collecting fresh examples. Otherwise, the system can improve on the test sheet without becoming more useful to the team.

For each run, retain the model identifier, prompt version, tool configuration, relevant data version, and outcome. These are what let you compare a change fairly.

Make the release decision explicit

At the pilot review, ask the owner to sign off on three decisions: which cases must pass, which requests still go to a human, and what will cause the pilot to stop.

A small test set will miss things. Pair it with limited exposure and review of real outcomes. Its immediate value is simpler: the team can name the behavior it is accepting and demonstrate that the next change has not quietly broken it.

That is a concrete deliverable to ask for when scoping an AI engagement.