Confidence is not calibrated
A language model’s tone carries no information about its accuracy. It will state an invented policy in the same register as a documented one, which means your users cannot self-defend by reading carefully. Every guardrail worth building starts from accepting that.
Prompting can reduce the rate of confident errors. It cannot make the remaining ones detectable, which is why prompt engineering is a poor substitute for system design.
Constrain the output space
The single most effective guardrail is refusing to let the model produce free text where a structured value will do. A classification into one of six categories can be validated; a paragraph explaining the classification cannot. Push as much of the feature as possible into shapes your code can check.
Ground answers in retrievable sources
Require the model to answer from supplied documents and to cite which one it used, then verify the citation exists and actually contains the claim. This turns an unfalsifiable assertion into a checkable one, and it fails loudly instead of silently when the source is missing.
You cannot stop a model from being wrong. You can stop it from being wrong invisibly.
Give it a supported way to decline
Models guess partly because the interface offers no alternative. An explicit “insufficient information” path, treated as a valid and unpunished outcome, converts a fabricated answer into a routed escalation. Log those escalations — they are a precise map of what your knowledge base is missing.
Escalate on stakes, not on confidence scores
Route to a human based on what the request is about — money, cancellations, legal, safety, anything irreversible — rather than on the model’s self-reported certainty. Stakes are a property of the request you can determine reliably; confidence is a property of the output you cannot.
This keeps your escalation policy stable across model upgrades, which matters more than it sounds when the underlying model changes twice a year.
Assume you will need to explain it
Log the inputs, the retrieved context, the output, and the human’s follow-up action for every interaction. The first serious complaint will require you to reconstruct exactly what happened, and “we think the prompt was roughly this” is not an answer anyone accepts.