What Happens When an AI Workflow Gets Something Wrong?

What kinds of mistakes does an AI workflow actually make?

An AI workflow rarely fails the way a machine fails. It does not stop with a red light. It keeps running, produces output that looks like every other day’s, and the mistake travels downstream into a ledger, an inbox or a customer’s expectation. So the useful question is not whether your system will get something wrong. It is how you will find out, how far it will have travelled by then, and how you will put it right.

Six failures cover most of what goes wrong in practice. They need different responses, so it helps to name them before you need to.

The first four produce a bad result. The fifth produces no result, which is easy to spot and easy to miss over a long weekend. The sixth costs the most, because nothing reports an error at all: a model version changes, a supplier alters a form, a price list moves, and the workflow carries on with quietly worse judgement.

How would you find out before the customer does?

Most teams find out from the person who was let down. That is a design choice, even if nobody made it deliberately. Decide instead what would tell you first, and put a number and an alert behind it. A daily count and a weekly sample do most of the work; our guide to measuring whether an AI system is working covers the measures themselves.

• Volume: how many documents, replies or records the workflow completed today, with an alert when the number falls to zero or doubles.
• Exception rate: how often the system asks for help. A sudden drop can mean it has stopped noticing what it does not know.
• Correction rate: how often a reviewer edits or rejects before approving, counted weekly rather than felt.
• Sampling: a person checks a small random sample of finished work each week, including work nobody complained about.
• Complaints tagged back to the workflow, so a pattern is visible instead of scattered across inboxes.
• Cost per completed task: a jump usually means retries or a loop nobody intended.

What does pausing an AI workflow safely look like?

A pause switch is a design decision, not a button someone improvises at 9 pm. Decide in advance what pausing means for each workflow, because stopping everything is sometimes worse than the fault: if the enquiry system stops, customers hear nothing at all.

• Narrow: switch to drafts only. The system still prepares the work; a person releases every item.
• Partial: stop one document type, one supplier, one branch or one customer, and let the rest run.
• Full: stop the workflow and route the incoming work to the manual process it replaced.

Then name who may pull it. The reviewer on shift should be able to narrow a workflow without finding the operations head first, and should never be blamed for stopping something that turned out to be fine. Write the fallback beside it: who does this work by hand meanwhile, and for how long that is bearable. The NIST AI Risk Management Framework asks for the same thing, calling for mechanisms and assigned responsibilities to supersede, disengage or deactivate systems behaving inconsistently with their intended use.

How do you contain the damage and correct the records?

Containment stops the mistake spreading; correction puts right what already spread. Both start with the step teams skip: work out how far it went before fixing anything.

1. Pause at the agreed level and tell the team, so nobody re-runs the job to see whether it works now.
2. Set the window. When did the fault start? The change log usually answers: a model version, an edited prompt, a new form layout, a changed permission.
3. List every record, message and decision produced inside that window. The list, not a sample.
4. Sort by consequence: money moved, promises made to customers, master records changed, reports already used.
5. Correct in that order, with a person approving each correction, and keep the original output as evidence.
6. Only then resume, watching the first day more closely than usual.

What do you tell the customers and colleagues affected?

Tell them early, in your own words, before they work it out. A short message beats a careful one that arrives three days later. Say what happened, what it affected, what you have already done, what they need to do, if anything, and who they can speak to. Tell your own team first, so nobody is caught unaware on a call, and record who was told, when and what was said. MeitY’s India AI Governance Guidelines make the same point from the other side: organisations deploying AI should run grievance redressal that is easy to reach and honours clear timelines, with the feedback analysed rather than filed.

When the failure exposes personal data rather than simply mishandling a record, a legal duty may also apply. Under India’s Digital Personal Data Protection Act, 2023, an organisation that decides how personal data is processed must protect it with reasonable security safeguards, and on a personal data breach must give the Data Protection Board and each affected person an intimation of it. The Digital Personal Data Protection Rules, 2025, notified in November 2025, set out what the person is told, without delay and in clear and plain language: a description of the breach, the consequences likely to matter to her, what you have done to reduce the risk, what she can do to protect herself, and contact details for someone who can answer her questions. The Board is told without delay, with fuller detail within seventy-two hours of your becoming aware.

How do you find the real cause and stop it repeating?

The instinct after a mistake is to correct the output, re-run the job and move on. That is how the same failure returns in six weeks with a different customer attached. Spend twenty minutes naming the cause instead, specifically enough that the fix is obvious:

• Input: the document, message or record was different from anything the system had seen.
• Source: the answer was right for a price list, policy or catalogue that nobody had updated.
• Instruction: the rule or prompt did not cover this case, or covered it ambiguously.
• Model: the provider changed a version, and behaviour changed with it.
• Integration: a connection failed, timed out or returned something unexpected.
• Permission: the system could reach or change something it should never have been able to.
• Process: a person approved without looking, usually because the gate asks for a decision nobody can make in the time given.

Write the cause in one plain sentence, decide one change, and give it an owner and a date. Then add the case to the evaluation set so the next release has to pass it. That last step is what turns an incident into an improvement, and it is why evaluation built from real business cases is worth the effort of maintaining.

What should an incident runbook contain?

A runbook is one page for each workflow that matters, written while everyone is calm. It says who acts at each stage and what gets written down, so the response does not depend on who happens to be in the office. Keep names, phone numbers and a date when it was last checked.

Rehearse it once, on a quiet afternoon, with an invented failure. Rehearsal is where you discover that only one person can pause the workflow and she is on leave, or that nobody can produce a list of everything the system did last Tuesday. A runbook first read during an incident is a document, not a control.

How do you design for recoverability before launch?

Recovery is cheaper to design in than to add afterwards, and most of it is plumbing agreed before the first release:

• Stage consequential actions: prepare, then release on approval, so a mistake is caught before it leaves the building.
• Prefer reversible actions: a draft rather than a send, a credit note rather than an edit, an archive rather than a deletion.
• Set batch and rate limits, so a loop cannot reach two thousand customers before anyone wakes up.
• Make repeated runs safe: the same input should produce one result, however many times a retry fires.
• Keep logs of the input, the sources used, the proposed action, the approver and the result, including failures.

None of this makes a system infallible. It is the difference between an hour of annoyance and a fortnight of reconstruction. Failure handling, named owners and cost visibility are part of how we work on every build, and our Infrastructure & Governance work covers detection, containment and recovery for systems already running.