That Time Client Retries Turned a Recovery Into a 7-Hour Outage

GitHub had an interesting incident last August. A component in Central US failed under load, and when it started recovering, the recovery took much longer than it should have. The culprit: clients hammering the recovering system with retries.

What happened

During the outage, GitHub's auth system was down. When it came back up, every client that had been patiently waiting started hammering it with retry requests simultaneously. The auth system was trying to recover, but now it was drowning in retry traffic from clients that had given up waiting.

This is the retry-loop trap. You've probably seen it in various forms:

• A health check endpoint that returns 503, so clients retry every second
• A connection pool that overflows, returns an error, clients reconnect immediately
• A CDN origin that's down, all edge nodes retry at the same rate

The pattern

The typical flow:

1. System goes down or slow
2. Clients start retrying with some interval
3. System partially recovers
4. Retries flood the recovering system
5. System goes down again or stays slow
6. Repeat until someone intervenes

What actually helps

Exponential backoff with jitter is the standard answer. Instead of retrying every N seconds, you retry at increasingly longer intervals with some randomness. Like: 1s, 2s, 4s, 8s, 16s... with +/- 20% jitter. This spreads out the retry load.

Circuit breakers stop forwarding requests to a failing service entirely for a cooldown period. The client knows the circuit is "open" and fails fast instead of retrying.

Rate limiting on the server side helps, but during recovery you often want to gradually increase capacity, not hard-cutoff.

Client-side rate limiting is underused. Most HTTP clients will retry indefinitely by default.

The GitHub specific angle

GitHub's postmortem mentioned record traffic that day - 115M Actions runs, 2.9B monthly commits. That's a lot of automated systems hammering retry loops. The lesson isn't that GitHub's infra was bad; it's that the combination of a brief outage + automated clients with naive retry logic creates a thundering herd on recovery.

If you're running a service that clients depend on, it's worth thinking about what their retry behavior looks like. Are they using exponential backoff? Do they have a circuit breaker? Or are they just spinning on a 1-second loop?

And if you're the client, it's worth checking your default retry config. The defaults in many HTTP libraries are not kind to recovering services.

This was the GitHub Aug 17 outage lesson. Source: DevOps'ish #323 and GitHub CTO Vlad Fedorov's postmortem.