AI features fail differently under load
Ordinary endpoints degrade gradually. Model calls hit provider rate limits and start being rejected outright, while each request holds a connection open for seconds — so a modest spike causes disproportionate failure.
Queue what doesn’t need to be immediate
Summarising, classifying and drafting rarely need to happen within a request. Moving them to a queue smooths spikes and lets you control concurrency against the provider’s limits rather than discovering them.
Retrying into a rate limit is how a small spike becomes an outage.
Back off properly
Naive retries multiply load exactly when the provider is refusing it. Exponential backoff with a cap and a maximum attempt count is essential — and retries must be counted separately from successes or your metrics will lie to you.
Decide what degrades
Under pressure, some features should stop so the important ones continue. Making that explicit beforehand is the same exercise as deciding what happens when the provider is unavailable.
Cache the repeated work
Identical questions arrive constantly, particularly in support. Caching answers to repeated inputs cuts cost and latency at once, and it is usually the cheapest capacity win available.
Cost spikes with traffic
Unlike servers, per-call pricing means a traffic surge is immediately a bill. A spending cap and an alert on unusual volume protect against both a viral moment and an abusive one — the cost-per-outcome discipline at the infrastructure layer.