AI features decay differently
Ordinary code keeps doing what it did. An AI feature degrades as the model changes beneath it, as its knowledge ages, and as the prompt accumulates corrections — none of which show up as an error.
1. Run the evaluation on a schedule
Not only on change. A scheduled run is the only thing that catches a provider changing the model underneath you, and it is the backbone of every other commitment here.
2. Keep the knowledge current
Business rules change and the documents behind the feature go stale, at which point it confidently repeats last year’s policy. Someone owns this, or it silently stops being true.
The failure mode isn’t an outage. It’s a system that keeps answering, using last year’s facts.
3. Prune the prompt
Instructions accumulate and nothing gets removed, which is why features get slower and more expensive after launch. Deleting safely requires version control and an eval to check against.
4. Watch cost per successful outcome
Not cost per call — per outcome, so retries, abandonment and rework are visible. Reviewed monthly, it catches drift long before finance does.
Budget the maintenance honestly
These four are recurring staff time, and they are most of what an AI feature costs over three years. Presenting a build cost without them is the omission that makes boards distrust the next proposal.