Today I Learned: There Are Three Kinds of Jitter
Whenever I hear “thundering herd,” I picture actual horses. The real thing is less scenic: a server goes down, thousands of clients notice at the same moment, and they all try to reconnect at the same moment too. Jitter is the standard fix, and today I learned there’s more than one way to calculate it.
Where the herd comes from
A dependency blips for two seconds. Every client with an open connection sees an error at roughly the same instant. If each one retries immediately, the server comes back to a wall of requests far bigger than its normal load, falls over again, and the cycle repeats.
The name is older than distributed systems. It originally described operating system processes all blocked on the same event: the event fires, every process wakes up, and only one of them gets to do anything. Same shape, bigger scale.
Backoff alone isn’t enough
The usual first fix is exponential backoff. Wait 100ms, then 200ms, then 400ms, doubling up to a cap.
```ruby BASE = 0.1 # seconds CAP = 30.0 # seconds
def backoff(attempt) [CAP, BASE * 2**attempt].min end ```
Backoff changes how often each client retries. It does nothing about when. A thousand clients that failed at the same moment all wait exactly 100ms, retry together, fail together, and then all wait exactly 200ms. The stampedes get further apart, and every one of them is still the whole herd.
Jitter adds randomness to the wait, so the clients spread out across the window and stop landing on the same millisecond.
Three ways to add jitter
The write-up most people point to is Marc Brooker’s Exponential Backoff And Jitter on the AWS Architecture Blog. It compares three approaches.
Full Jitter picks a random wait anywhere from zero up to the backoff value.
ruby
def full_jitter(attempt)
rand(0.0..backoff(attempt))
end
Equal Jitter keeps half of the backoff and randomizes the other half, so every client waits at least some minimum.
ruby
def equal_jitter(attempt)
half = backoff(attempt) / 2
half + rand(0.0..half)
end
Decorrelated Jitter ignores the attempt count. Each wait is a random value between the base delay and three times the previous wait, capped.
ruby
def decorrelated_jitter(previous_sleep)
[CAP, rand(BASE..previous_sleep * 3)].min
end
Start it with BASE as the previous sleep, and feed each result into the next call.
Picking one
Each one trades something. Full Jitter spreads clients the widest, but some unlucky client will retry almost immediately. Equal Jitter guarantees a floor, at the cost of bunching everyone into the upper half of the window. Decorrelated Jitter lets each client’s delay wander based on its own history, so two clients that started in lockstep drift apart quickly.
In Brooker’s simulations, all three jittered versions did far less total work than plain backoff with no jitter. That’s the part I’m holding onto. Which formula you pick matters a lot less than picking one.
If I had to choose a default, I’d take Full Jitter. It’s one line, and it spreads the herd the widest.
A few things around the formula matter just as much:
- Cap the delay, or a client that’s been failing for an hour ends up waiting most of a day between attempts.
- Cap the number of retries too. Every retry is extra load on a system that’s already struggling, and at some point the right move is to give up and surface the error.
- Retry only what’s safe to retry. A
GETis fine. APOSTthat charges a credit card needs an idempotency key before it gets anywhere near a retry loop.
Next time I picture the horses, I’ll picture them leaving the barn a few at a time.