Worker Tuning
Tune the queue consumers and import jobs you own. Raise one control at a time and stop where useful throughput stops improving.
Find the point where throughput stops paying
Run the same workload at two concurrency settings and enter what you measured. Scaling efficiency is the throughput gain divided by the concurrency increase — anything well under 100% means you are adding workers without adding completed work.
Still scaling well. Keep raising concurrency one step at a time.
Keep the payload mix and batch size fixed while you do this, or you are comparing two different workloads. Settle one step below the knee so normal variance and interactive traffic still have room.
Give every job a terminal outcome
- Complete
- Acknowledge only after the API confirmed the write and your side-effects are durable.
- Retry
- Transient failures and 429 responses, with backoff, jitter, and a hard attempt cap.
- Quarantine
- Permanent failures leave the hot queue with the payload preserved for inspection.
- Never silent
- No job may disappear without landing in one of the states above.
Survive deploys and restarts
The riskiest moment is a worker stopping between a successful API call and its queue acknowledgement. Decide what a redelivered job means for your data, then prove it: stop a worker mid-flight in a test environment and confirm the queue drains to exactly one intended outcome.