Plan for the incident, not the average
Every automated system has a worst day. The question is not whether an AI employee will tell a customer something untrue, but whether you find out quickly, can explain what happened, and can make it right before it spreads.
You need the transcript, complete
The first thing anyone asks is what exactly it said. If you cannot reconstruct the conversation, the retrieved context, and the version of the rules in force at the time, you are guessing in front of an unhappy customer — which is why logging inputs and decisions rather than just outcomes matters before you need it.
Own it plainly
The recovery that works is the ordinary one: acknowledge the error, correct it specifically, and say what changed so it won’t recur. Attempting to obscure that an automated system was involved converts a service failure into a trust failure, which is a far more expensive category.
Customers forgive a wrong answer. They don’t forgive discovering you hid how it was produced.
Fix the class, not the instance
Patching the specific case teaches you nothing. Ask what category of question produced it, whether the knowledge was missing or contradictory, and whether the request should have triggered an escalation on stakes rather than been answered at all.
Then add it to the evaluation
Every real incident is the highest-quality test case you will ever get, because it actually happened. Adding it to your eval set is what converts an embarrassment into a permanent guarantee it won’t repeat silently.
A team that does this consistently finds its incident rate falls even as scope grows. A team that patches instances finds the same failure returns in a new costume.