Production / Traffic control
Horizontal Scaling
More application instances multiply the concurrency you send, not the capacity you are granted. Size the per-instance limit deliberately and ramp against gates.
Sizing model
Size the per-instance budget
Concurrency is throughput multiplied by service time. Work out the per-instance ceiling deliberately, then set it as configuration — an unbounded client is how a routine deploy turns into a self-inflicted traffic spike.
Per-instance RPS
20.0
Per-instance concurrency cap
6
Fleet in-flight ceiling
36
The fleet ceiling is the number that matters to the API. If it looks surprising, reduce the per-instance cap before adding instances, and keep interactive traffic on a separate budget from batch work.
Ramp
Ramp in steps, not in one jump
| Stage | Share of target | Continue only if |
|---|---|---|
| Warm | 10% | No 429s and latency inside your budget |
| Half | 50% | Error rate flat and queue age stable |
| Target | 100% | Latency and quality unchanged for a full window |
| Headroom | 130% | Failure behavior still degrades gracefully |
Stop conditions
Signals that mean stop adding load
- Throughput stops rising while attempts keep rising — you are measuring retries, not capacity.
- The oldest item in your queue keeps getting older across cycles.
- 429 responses appear in normal operation rather than only during bursts.
- Latency grows on the client while payload sizes and query shapes are unchanged.
Was this page helpful?