The dependency that changes itself
Model providers improve, deprecate, and retire versions on their own schedule. Your AI employee can start behaving differently on a day you shipped nothing, which is an unusual property for a dependency to have.
Better is not the same as compatible
A newer model may be stronger in general and worse at your specific task — more verbose, differently formatted, less willing to follow one of your instructions. Only your own evaluation on your own inputs can tell you which.
An upgrade you didn’t ask for is still a change you have to test.
Run the eval on a schedule, not on change
Triggering tests only when you edit something misses upstream shifts entirely. A scheduled run is what converts a silent provider change into a visible signal — the monitoring half of building on someone else’s model.
Pin versions where the provider allows it
Explicit version pinning turns an unexpected change into a deliberate migration you schedule. Combined with deprecation notices, it converts a surprise into a task with a date.
Keep the switch behind your own interface
Moving to a new version or provider should be a configuration change, which is only true if every call already goes through one module. Retrofitting that under time pressure is the expensive path.
Tell the reviewer when it happens
Whoever owns the AI employee should raise the sampling rate for a fortnight after any model change. That is the window in which subtle regressions show up in real work rather than in tests.