AlgoInsight

System Design

Designing a Distributed Rate Limiter That Survives a Traffic Spike

Every candidate can build a counter. Then I add one sentence - 'now run it across fifty servers' - and the simple version quietly falls apart.

By Admin2026-06-1811 min readrate-limitingdistributed-systemssystem-designtoken-bucketredisscalability

Years ago, before I learned what "scale" actually meant, I watched an API endpoint melt. A single client - buggy, not malicious - got stuck in a tight loop and started firing thousands of requests a second at a service that was never built to absorb it. The database saturated, latency climbed, and requests from every other customer started timing out behind the flood. One misbehaving caller took down the experience for everyone.

The thing that should have stood between that client and the database is a rate limiter. It's one of the most common system design questions I ask, because it looks easy and isn't. Almost everyone can build the single-server version. The interview - and production - is about what happens when there's more than one server. Let me build it up the way I wish someone had built it up for me.

Why rate limit at all

A rate limiter caps how many requests a client can make in a window of time. It exists to protect you from four things:

The output is simple: for each incoming request, the limiter says allow or deny (usually a 429 Too Many Requests). All the difficulty is in how it decides.

The algorithms, and the flaw hiding in the obvious one

Fixed window is what everyone invents first. Count requests per client per fixed clock window - say, 100 per minute. At the top of each minute, reset the count to zero.

It's simple and it's wrong at the edges. Picture a limit of 100/minute. A client sends 100 requests at 10:00:59, then 100 more at 10:01:00. Both windows are individually legal - but in that one real-world second around the boundary, you let through 200 requests. The fixed window can permit a burst of double your limit right at the seam.

flowchart TD
    R[Request arrives] --> A{Tokens available<br/>in the bucket?}
    A -->|Yes| B[Remove 1 token]
    B --> C[Allow request ✓]
    A -->|No| D[Deny request → 429 ✗]
    E((Refill clock)) -.adds tokens<br/>at a steady rate.-> A

This flowchart shows a token bucket rate limiter. A request arrives and the limiter checks whether tokens are available in the bucket. If yes, it removes one token and allows the request. If no, it denies the request with a 429 error. Separately, a refill clock continuously adds tokens back into the bucket at a steady rate.

The fixes climb a ladder of sophistication:

Here's a token bucket you can run:

import time

class TokenBucket:
    def __init__(self, capacity, refill_per_sec):
        self.capacity = capacity          # max burst
        self.tokens = capacity
        self.refill = refill_per_sec      # steady-state rate
        self.last = time.monotonic()

    def allow(self):
        now = time.monotonic()
        # add tokens for the time elapsed, capped at capacity
        self.tokens = min(self.capacity, self.tokens + (now - self.last) * self.refill)
        self.last = now
        if self.tokens >= 1:
            self.tokens -= 1
            return True
        return False

bucket = TokenBucket(capacity=5, refill_per_sec=2)
results = [bucket.allow() for _ in range(8)]   # 8 requests, all at once
print(results)
print(f"Allowed: {sum(results)}/8  (burst of 5, then throttled)")

The first five requests sail through - that's the burst the bucket allows. The rest are denied until the refill clock tops the bucket back up. Steady average, controlled burst. That's exactly the behavior you want.

Now the real question: make it distributed

Everything above lives happily on one server. But you don't run one server. You run fifty, behind a load balancer, and a client's requests scatter randomly across all of them. Now the simple design breaks in a way that's easy to miss.

If each of your fifty servers keeps its own token bucket, your real limit isn't 100/minute - it's 100 × 50 = 5,000/minute, because the client can spend a full bucket on every server independently. Your carefully-designed limiter is off by a factor of fifty and you might not notice until someone abuses it.

Important: The core challenge of a distributed rate limiter is shared state. All your servers must agree on a single, authoritative count for each client. The instant the counter lives in per-server memory, the limit becomes meaningless across the fleet.

The standard answer: move the counter to a fast, centralized store that every server can reach - almost always Redis, because it's in-memory fast and supports atomic operations. Every server, on every request, asks Redis: "decrement this client's tokens; is there one left?"

Which surfaces the next subtlety - the race condition. If two servers read the same client's count at the same moment, both see "1 token left," and both allow. You've leaked a request past the limit. The fix is to make the read-modify-write atomic: a single Redis operation (an atomic INCR, or a small Lua script that checks and decrements in one indivisible step) so two requests can never both win the last token.

Interview: When I ask "design a rate limiter," the single sentence that separates senior from mid-level is: "I'll keep the counter in Redis and make the check-and-decrement atomic with a Lua script, so concurrent requests across servers can't both consume the last token." That one line tells me you've actually thought past the single-machine version. Most candidates never get there on their own - I have to drag them.

The trade-offs you should name out loud

There's no perfect distributed limiter; there are positions on a spectrum, and naming the trade is the whole game:

Concern The tension
Accuracy vs latency A perfectly-accurate global count means every request makes a network round-trip to Redis. That adds latency to every call. Some systems accept slightly fuzzy limits to avoid that hop.
Centralized vs local A central Redis is accurate but is itself a dependency (and a potential bottleneck/SPOF). Some designs let each server enforce a local share of the limit - fast, but approximate.
Fail-open vs fail-closed If Redis is unreachable, do you allow all requests (fail-open: protect availability, lose enforcement) or deny them (fail-closed: protect the backend, hurt users)? There's no universal right answer - it depends on what you're protecting.

That last one is a favorite of mine because it has no clean answer. For a public content API I might fail-open; the cost of over-limiting real users is worse than briefly under-protecting. For an expensive payment or AI-inference endpoint I'd fail-closed; I'd rather reject traffic than let an outage turn into a runaway bill.

The candidates who clear my loop on this question aren't the ones who memorized four algorithms. They're the ones who build the single-server token bucket calmly, and then - before I even ask - say "but across a fleet this needs shared atomic state, so here's how I'd use Redis, and here's the fail-open-vs-fail-closed call I'd make." That progression is the skill. We drill exactly this kind of build-it-up reasoning across System Design Fundamentals.

Go deeper

More articles