Rate limits and quotas: why your AI feature stopped working
Why AI features fail when a provider's limits or caps are hit, and how to design so users see graceful behaviour instead of errors.
The short answer
Every AI provider limits how much an account may use: requests per minute, tokens per minute, requests per day, concurrent requests, and a spending cap per month, with different limits at different account tiers. When a feature crosses one, the provider rejects requests until the window resets, and a feature that worked perfectly in testing returns errors to every user at once. The trigger is usually ordinary: usage grew, a scheduled batch job ran, a campaign brought traffic, or a bug looped. The failure is predictable, and so is the design that prevents it from reaching users: retries with increasing delays for transient rejections, queues for batch work so it absorbs limits invisibly, graceful fallbacks for interactive features, monitoring of usage against limits with alerts well before they are hit, and your own caps set below the provider’s so a bug or an attack cannot run up a bill. Plan capacity from measured usage, request higher limits before you need them, and never let a spending cap be discovered on the busiest day of the quarter.
The limits and what hits them
| Limit | What it caps | Typical trigger | Design response |
|---|---|---|---|
| Requests per minute | How many calls in a minute | Traffic spike; parallel batch jobs | Retry with backoff; queue; spread batch work |
| Tokens per minute | How much text in and out per minute | Long documents; large batches | Chunking; queue; smaller context |
| Requests per day | Daily volume | Growth; a new integration | Monitor against the cap; request increase |
| Concurrent requests | How many in flight at once | Parallel processing | Concurrency limits in your code |
| Spending cap | Money per month | Growth; a loop; an attack on a public feature | Your own lower cap; alerts at thresholds; rate limits on public endpoints |
| Context length | Size of a single request | A very long document | Chunking and summarisation |
| Model-specific limits | Newer or larger models have tighter limits | Switching models | Check limits before switching; fall back to another model |
Designing for limits
- Know the limits on your account for each model you use, and write them down.
- Set your own caps below them: per user, per feature, per day, and a spending alert at a fraction of the monthly cap.
- Retry with backoff on transient rejections, honouring the provider’s retry guidance.
- Queue batch work so it flows at a controlled rate and resumes after a limit resets.
- Fall back gracefully in interactive features: a simpler response, a cached answer, a clear message.
- Rate-limit public endpoints that call AI, so one visitor or a bot cannot consume the account’s capacity.
- Monitor usage against limits and alert at thresholds, with the trend visible.
- Request increases ahead of need, with the usage data to support them.
- Have a second model or provider configured for critical paths.
Planning capacity
Measure actual usage per feature per day and its growth. Model the peaks: what a campaign or a batch does to the per-minute numbers. Compare with the limits and the caps. Request increases when projected peaks approach limits, not when they exceed them. And revisit monthly, because AI usage in a business tends to grow faster than anyone planned once a feature works.
What this means for you
AI features stop working when provider limits or spending caps are hit, and they will be hit as usage grows. Build for it: retries with backoff, queues for batch work, graceful fallbacks for interactive use, rate limits on public endpoints, your own caps below the provider’s, monitoring with alerts, and limit increases requested from data ahead of need. Then a limit is a delay nobody notices rather than an outage everybody does.
Frequently asked questions
Our AI feature worked for weeks and then suddenly stopped. Why?
Most often because usage crossed a provider limit: requests per minute, tokens per minute, requests per day, or a spending cap on the account. Growth, a scheduled batch job, a marketing push or a runaway loop pushed usage over the line, the provider began rejecting requests, and the feature had no handling for rejection, so users saw errors. The fix has two parts: handle the rejections gracefully, and monitor usage so the limit is raised or the load is shaped before it is hit.
Should we just ask for higher limits?
Yes, ahead of need, with usage data, because providers raise limits based on history and account standing and it takes time. But higher limits are not a design. The feature still needs retry logic, queuing for non-urgent work, fallbacks for interactive use, and your own caps so a bug or an attack cannot run up a bill. Limits will be hit eventually; the question is what users see when they are.
What should users see when a limit is hit?
For interactive features: a graceful message, a fallback such as a simpler non-AI response, or a slight delay while the request retries, never a raw error. For batch work: nothing, because the queue absorbs it and processing resumes when the window resets. For internal tools: a clear status showing the backlog. The user's experience should degrade gently, and the team should be alerted before users notice anything.