How to Evaluate AI Quality Using Real Business Cases

Why do demonstrations pass and real work fail?

A demonstration uses the file somebody chose. It is legible, complete and typical of nothing. Real work arrives as a photograph of a delivery challan taken at an angle in poor light, an enquiry that asks two questions in two languages, an invoice from a supplier who changed their layout last month, and the same purchase order sent twice because the first email bounced. A system that handles the chosen file beautifully can still be wrong again and again on a normal Tuesday.

An evaluation set closes that gap. It is a fixed collection of real cases with the correct outcome written down for each, run the same way every time, so that a change in the score means a change in the system rather than a change in which examples someone tried. It is the difference between believing a system is good and being able to show it, and it is the only honest basis for the decision to release.

How do you build an evaluation set from real work?

Take the cases from the work itself, over a period long enough to include a month-end and at least one quiet week. Do not write cases from imagination; you will invent the problems you already know about and miss the ones that actually happen.

• Size: enough to show a pattern, few enough that a person can review the results in an afternoon. Forty to eighty cases per workflow is a workable start.
• Mix by frequency: if two-thirds of your invoices come from ten suppliers, the set should look like that, or your score will describe work you rarely do.
• Add the awkward ones deliberately: the poor scan, the message in three languages, the missing GSTIN, the duplicate, the supplier who changed their format, the customer who asks two things at once.
• Include cases where the correct outcome is to refuse, escalate or ask a question.
• Include a few cases from before the system existed, where you know how the business actually decided and why.
• Freeze it and version it. A set that quietly changes between runs cannot tell you whether anything improved.

What is the right answer when stopping is the right answer?

Every case needs an expected outcome written before the system sees it. Otherwise the review becomes a discussion of whether the output looks reasonable, which it usually does. What the expected outcome contains depends on the kind of work:

• Extraction: the exact values for each required field, and where in the document they came from.
• An answer: the facts that must appear, the facts that must not, and the source it should be drawn from.
• An action: what should change in the other system, and what should not change.
• Stop and ask: the case is ambiguous, incomplete or outside the agreed limits, so the correct output is a question to a named person, with the reason.

That last category changes behaviour more than any other. If your set only rewards answers, you are training your reviewers and your suppliers to prefer a confident guess to an honest stop. Mark those cases clearly, and treat a system that answered them anyway as having failed, even when the answer happened to be right. Deciding where a workflow should pause is a design question in its own right, covered in designing the human decision into the workflow.

How do you score work that is not simply right or wrong?

Right and wrong is too blunt for most business work. An answer can be correct but missing a caveat, or correct with a small edit, or wrong in a way that would have cost money. A short rubric, agreed in advance and applied the same way by everyone, gives you a number you can compare between runs.

Weight the score by consequence rather than counting cases equally. A wrong pin code and a wrong bank account are not the same event. The NIST AI Risk Management Framework makes the same point in its own language: performance or assurance criteria should be measured and demonstrated for conditions similar to the setting the system will be deployed in, and the test sets and tools used should be documented.

How do you know your reviewers agree?

Two experienced people scoring the same case can reach different verdicts, particularly on the middle categories. If your quality figure depends on who happened to score the run, it cannot be compared with last month. So test the scorers before you trust the score: give two people the same twenty cases, independently, and compare.

Where they disagree, the rubric is unclear, not the reviewer. Rewrite the definition, add an example of each score, and run the twenty again. It is also worth involving someone who did not build the system. The NIST framework asks that internal experts who did not serve as front-line developers, or independent assessors, take part in regular assessments, with domain experts and users consulted as needed. In a smaller company, that can simply be the colleague who does the work every day rather than the person who wrote the instructions.

What is good enough to release, and when do you run it again?

Agree the pass level with the business owner before anyone sees the results, or the threshold will quietly become whatever the system scored. Express it in categories rather than a single average, because one number hides the failures that matter:

• No wrong and confident results at all on anything that moves money, changes a master record or commits you to a customer.
• An agreed, small number of incomplete results elsewhere, each with a route for a person to catch it.
• A ceiling on the edit rate, since a system that is right but always edited is costing the reviewer more than it saves.
• A floor on the stop-and-ask cases: they must be handled correctly, not answered anyway.

Then re-run the set after every change: a revised prompt, a new source, a new document type, a changed integration, and above all a new model version, including one your provider deploys without asking you. Re-run on a schedule as well, perhaps quarterly, even when nothing on your side changed, because suppliers, formats and customers change under you. Keep each result dated and tied to the version it describes. Evaluation, logs and cost visibility are the everyday work of running a system, which is what our Infrastructure & Governance practice sets up; if you are still deciding which workflow deserves a build, the AI Opportunity Blueprint is a paid diagnostic that maps the work and assesses readiness first.

How do you stop test cases leaking, and protect the data in them?

There is a quiet way to fool yourself. A case fails, someone pastes it into the system’s instructions as an example of how to handle that situation, and the next run passes. You have not taught the system to reason; you have told it the answer. Keep the examples inside the prompt separate from the cases you score, and hold back a portion of the set that the people writing instructions never see. When a failure teaches you something general, write the general rule and add a fresh, similar case to the set rather than the one you fixed.

Evaluation sets are made of real customer records, so treat them as such. Under India’s Digital Personal Data Protection Act, 2023, an organisation must protect personal data in its possession with reasonable security safeguards and erase it, unless a law requires retention, once it is reasonable to assume the purpose is no longer being served. The Digital Personal Data Protection Rules, 2025 describe those safeguards in practical terms: measures such as encryption, masking or tokens, control over who can reach the data, and visibility of access through logs, monitoring and review.

In practice that means a few habits. Keep the set inside systems you already control, not on a laptop or a personal drive. Mask what the test does not need, since a name and an account number rarely affect whether a total was read correctly. Keep the list of who can open it short, and note where the cases came from and when they should be removed. Where a case is too sensitive to copy, test against it in place.