By Gaurav · 2026-11-01 · 21 min read

• distributed-systems
• backend
• redis
• reliability
• algorithms

Rate limiting looks like a small feature. You count requests, compare the count to a number, and return 429 when the count is too high. A junior engineer can implement that in an afternoon, and it will pass every test in a single-process environment.

Then the service runs on twelve pods behind a load balancer, each pod keeps its own counter, and the effective limit becomes twelve times what you configured. Or the counter lives in Redis, and two requests read the same value, both decide they are under the limit, and both increment — a classic lost update. Or the limit is correct but the retry policy turns a brief spike into a sustained outage. Or the limiter itself becomes the bottleneck and takes down the service it was supposed to protect.

This article is about the gap between the textbook algorithm and the production system. It covers the main algorithms and what they actually guarantee, the race conditions that appear when state is shared, how to key limits so they protect the right thing, what a correct 429 response looks like, and how to design a limiter that degrades safely when it is under stress. The JavaScript examples are runnable and use a fake clock, so you can inspect the behavior instead of trusting a diagram.

What rate limiting is for

Before choosing an algorithm, be clear about the goal. "Rate limiting" is used for at least five different jobs, and they pull the design in different directions:

1. Protecting a scarce resource. A database can handle 2,000 queries per second; a third-party API allows 100 calls per minute; an LLM endpoint costs money per token. The limit exists to keep demand below capacity.
2. Fairness. One tenant should not consume the whole service. Per-tenant limits ensure that a noisy neighbor cannot starve everyone else.
3. Abuse prevention. Credential stuffing, scraping, and enumeration attacks are characterized by high request rates from few sources. Limits raise the cost of the attack.
4. Cost control. Cloud services bill per request, per byte, or per token. A limit is a budget enforcement mechanism.
5. Backpressure signaling. A 429 tells a well-behaved client to slow down, which is cheaper than letting it time out or fail in a more expensive way.

These goals imply different keys, different thresholds, and different failure modes. An abuse limiter keyed by IP wants to fail closed and tolerate false positives. A cost-control limiter keyed by tenant wants to be accurate and observable, because someone will ask why their bill looks the way it does. A capacity-protection limiter wants to shed load before the database melts, which means it should react to the database's actual health, not just to a request count.

There is also an important distinction between rate, concurrency, and quota:

• A rate limit bounds requests per unit time: 100 requests per second.
• A concurrency limit bounds requests in flight at once: 20 simultaneous requests. This is often the better protection for slow downstreams, because it directly bounds queue depth and memory.
• A quota bounds total usage over a long period: 1 million requests per month. It is usually enforced with a counter, not a token bucket.

A service often needs all three. Rate limiting alone does not prevent 20 slow requests from occupying every worker for a minute. Concurrency limiting alone does not prevent a fast client from making 100,000 cheap requests. Quota alone does not protect against a burst.

Token bucket: the default mental model

The token bucket is the most useful general-purpose algorithm. A bucket holds up to capacity tokens and refills at rate tokens per second. Each request removes one token (or more, for weighted requests). If the bucket is empty, the request is rejected or queued.

It has two parameters with distinct meanings:

• capacity controls burst size. A bucket that holds 100 tokens allows a burst of 100 requests after an idle period.
• rate controls the long-run average. Over any sufficiently long interval, throughput approaches rate.

That separation is why the token bucket is so widely used. It permits short bursts, which real traffic has, while enforcing a sustainable average. A fixed window does not give you that control: it permits a burst at the boundary regardless of recent history.

Here is a deterministic implementation using an injected clock:

class TokenBucket {
  constructor(capacity, ratePerSecond, now = 0) {
    this.capacity = capacity;
    this.rate = ratePerSecond;
    this.tokens = capacity;
    this.last = now;
  }

  refill(now) {
    const elapsed = Math.max(0, now - this.last) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.rate);
    this.last = now;
  }

  tryRemove(count = 1, now = this.last) {
    this.refill(now);
    if (this.tokens >= count) {
      this.tokens -= count;
      return true;
    }
    return false;
  }

  retryAfterMs(now = this.last, count = 1) {
    this.refill(now);
    const missing = count - this.tokens;
    if (missing <= 0) return 0;
    return Math.ceil((missing / this.rate) * 1000);
  }
}

const bucket = new TokenBucket(5, 2, 0);

// Burst of five succeeds.
for (let i = 0; i < 5; i += 1) {
  console.log(`t=0 request ${i + 1}: ${bucket.tryRemove(1, 0)}`);
}

// The sixth is rejected, and we can tell the client when to retry.
console.log(`t=0 request 6: ${bucket.tryRemove(1, 0)}`);
console.log(`retry after: ${bucket.retryAfterMs(0)} ms`);

// After 1.5 seconds, three tokens have refilled.
console.log(`t=1500 request 7: ${bucket.tryRemove(1, 1500)}`);
console.log(`tokens left: ${bucket.tokens.toFixed(2)}`);

The implementation stores only two numbers: the current token count and the timestamp of the last refill. Lazy refill — computing the new token count when the next request arrives — is what makes this cheap. There is no background timer per bucket, which matters when you have millions of keys.

Two details are easy to get wrong. First, cap the refill at capacity; otherwise an idle bucket accumulates unlimited credit and the next burst is unbounded. Second, use a monotonic clock where possible. Date.now() can jump backward during an NTP correction, which would make elapsed negative and either create or destroy tokens depending on how you handle it. The implementation above clamps elapsed at zero, which is a safe fallback.

Weighted requests

Not all requests cost the same. A search query might cost 5 units, a cheap health check 1, and a report generation 100. A token bucket handles this naturally: remove cost tokens instead of one. The bucket then enforces a cost rate, which is usually closer to what you actually want to protect.

const costs = {
  health: 1,
  search: 5,
  report: 50,
};

class WeightedBucket {
  constructor(capacity, rate, now = 0) {
    this.capacity = capacity;
    this.rate = rate;
    this.tokens = capacity;
    this.last = now;
  }

  take(operation, now) {
    const cost = costs[operation] ?? 1;
    const elapsed = Math.max(0, now - this.last) / 1000;
    this.tokens = Math.min(this.capacity, this.tokens + elapsed * this.rate);
    this.last = now;

    if (this.tokens >= cost) {
      this.tokens -= cost;
      return { allowed: true, remaining: Math.floor(this.tokens) };
    }

    return { allowed: false, remaining: Math.floor(this.tokens), cost };
  }
}

const bucket = new WeightedBucket(60, 10, 0);

console.log(bucket.take("health", 0));
console.log(bucket.take("search", 0));
console.log(bucket.take("report", 0));
console.log(bucket.take("report", 0));
console.log(bucket.take("report", 5000));

This is also the right way to think about LLM token limits. The "request" is not the unit; the input plus output tokens are. Weighting by estimated cost makes the limiter correspond to the bill and to the downstream load.

Leaky bucket and GCRA

The leaky bucket is the mirror image of the token bucket. Requests enter a queue of fixed size and drain at a fixed rate. If the queue is full, new requests are rejected. It smooths traffic rather than allowing bursts, which is useful when the downstream cannot tolerate bursts at all.

A queue-based leaky bucket requires storage per pending request, which is expensive for a limiter that must handle millions of keys. The Generic Cell Rate Algorithm (GCRA) achieves the same smoothing with a single timestamp per key. It is the algorithm behind many high-performance rate limiters, including some API gateways.

GCRA tracks a theoretical arrival time (TAT). Each conforming request advances the TAT by the emission interval. A request is conforming if its arrival time is at or after TAT - burstTolerance. The burst tolerance is expressed as a number of emission intervals.

class Gcra {
  constructor(ratePerSecond, burst, now = 0) {
    this.emissionIntervalMs = 1000 / ratePerSecond;
    this.burstMs = burst * this.emissionIntervalMs;
    this.tat = now;
  }

  check(now) {
    const earliest = this.tat - this.burstMs;
    if (now < earliest) {
      return {
        allowed: false,
        retryAfterMs: Math.ceil(earliest - now),
      };
    }

    // Advance the theoretical arrival time, but never let it fall
    // behind the current time by more than the burst tolerance.
    this.tat = Math.max(this.tat, now) + this.emissionIntervalMs;

    return { allowed: true, retryAfterMs: 0 };
  }
}

const limiter = new Gcra(10, 5, 0);

for (let i = 0; i < 8; i += 1) {
  const result = limiter.check(0);
  console.log(`burst request ${i + 1}:`, result);
}

// A request 250 ms later: the bucket has had time to recover.
console.log("after 250ms:", limiter.check(250));

GCRA and the token bucket are duals. The token bucket stores remaining credit; GCRA stores the next allowed time. Both produce the same long-run rate, and both allow a bounded burst. GCRA's advantage is that it needs one number instead of two and avoids floating-point token counts. Its disadvantage is that the parameters are less intuitive: burst is expressed in requests, and the relationship between TAT and wall-clock time takes a while to internalize.

The important operational property is that both algorithms can answer "when should the client retry?" Without that, a rejected client has no choice but to guess, and guessing produces synchronized retries.

Fixed windows and the boundary bug

The simplest limiter is a counter per time bucket: key user:42:2026-11-01T12:34, increment on each request, reject when the count exceeds the limit, expire the key after the window. It is cheap, easy to understand, and wrong in a way that users notice.

The fixed window permits up to twice the configured rate at a window boundary. If the limit is 100 per minute and a client sends 100 requests at 12:34:59 and another 100 at 12:35:00, all 200 succeed. The counter for the first minute is full, and the counter for the second minute is empty. From the server's perspective, nothing is violated. From the downstream's perspective, it just received 200 requests in one second.

You can demonstrate the accounting directly:

function fixedWindow(key, nowMs, limit, windowMs) {
  const windowStart = Math.floor(nowMs / windowMs) * windowMs;
  const bucketKey = `${key}:${windowStart}`;
  const count = (fixedWindow.state.get(bucketKey) ?? 0) + 1;
  fixedWindow.state.set(bucketKey, count);

  return {
    allowed: count <= limit,
    key: bucketKey,
    count,
    windowStart,
  };
}
fixedWindow.state = new Map();

const limit = 5;
const windowMs = 1000;

// Five requests at the end of one window...
for (let i = 0; i < 5; i += 1) {
  fixedWindow("user", 999, limit, windowMs);
}

// ...and five more just after the boundary.
for (let i = 0; i < 5; i += 1) {
  const result = fixedWindow("user", 1000, limit, windowMs);
  console.log(result);
}

Ten requests passed in roughly one millisecond of wall-clock time, against a limit of five per second. The algorithm is not broken; the model is too coarse. A fixed window measures the wrong thing.

Fixed windows are still used because they are cheap and because some quotas genuinely are per calendar window ("1,000 requests per day"). For those, the boundary behavior is acceptable or explicitly part of the contract. For burst protection, it is not.

Sliding window log and sliding window counter

The sliding window log stores the timestamp of every request in the window and counts them. It is exact: at any moment, it knows precisely how many requests occurred in the last N seconds.

class SlidingWindowLog {
  constructor(limit, windowMs) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.log = [];
  }

  check(now) {
    const cutoff = now - this.windowMs;
    while (this.log.length > 0 && this.log[0] <= cutoff) {
      this.log.shift();
    }

    if (this.log.length >= this.limit) {
      const retryAfterMs = this.log[0] + this.windowMs - now;
      return { allowed: false, retryAfterMs };
    }

    this.log.push(now);
    return { allowed: true, retryAfterMs: 0 };
  }
}

const limiter = new SlidingWindowLog(3, 1000);

for (const t of [0, 100, 200, 300, 1200, 1250]) {
  console.log(t, limiter.check(t));
}

The log is exact but memory grows with the limit. A limit of 10,000 requests per hour means up to 10,000 timestamps per key. At millions of keys, that is not viable. It is a good choice for expensive, low-volume operations — password resets, expensive reports, login attempts — where accuracy matters more than memory.

The sliding window counter approximates the log with two counters. It keeps the count for the current fixed window and the count for the previous window, then estimates the current rate by weighting the previous count by how much of its window overlaps the trailing interval.

class SlidingWindowCounter {
  constructor(limit, windowMs) {
    this.limit = limit;
    this.windowMs = windowMs;
    this.windows = new Map();
  }

  check(key, now) {
    const currentStart = Math.floor(now / this.windowMs) * this.windowMs;
    const previousStart = currentStart - this.windowMs;

    const current = this.windows.get(`${key}:${currentStart}`) ?? 0;
    const previous = this.windows.get(`${key}:${previousStart}`) ?? 0;

    const elapsed = now - currentStart;
    const previousWeight = (this.windowMs - elapsed) / this.windowMs;
    const estimated = previous * previousWeight + current;

    if (estimated >= this.limit) {
      return { allowed: false, estimated: Number(estimated.toFixed(2)) };
    }

    this.windows.set(`${key}:${currentStart}`, current + 1);
    return { allowed: true, estimated: Number(estimated.toFixed(2)) };
  }
}

const limiter = new SlidingWindowCounter(5, 1000);

// Fill the first window.
for (let i = 0; i < 5; i += 1) {
  limiter.check("user", 900);
}

// Just after the boundary, the previous window is weighted at ~100%.
console.log(limiter.check("user", 1000));
console.log(limiter.check("user", 1100));

// Halfway through the new window, the previous window's weight is halved.
console.log(limiter.check("user", 1500));

This is the algorithm behind many API gateways and Cloudflare-style edge limiters. It smooths the boundary spike and uses constant memory per key. The estimate is not exact: it assumes requests in the previous window were uniformly distributed, which they were not. Under a burst at the end of the previous window, it can still allow slightly more than the true sliding limit, but much less than the fixed window's 2x overshoot.

There is a real tradeoff here. Exactness costs memory. Constant memory costs accuracy. The right choice depends on what the limit protects. If the downstream will fall over at 2x the limit, use a log or a token bucket. If the limit is a fairness policy and a 20% overshoot is harmless, the counter is fine.

Key design is the actual product decision

The algorithm is the easy part. Deciding what to count is where most rate-limiting bugs live.

A limit key answers the question "who shares this budget?" Common choices:

The key must match the threat model. An IP-based limit is weak against distributed attacks and unfair to users behind carrier-grade NAT. A user-based limit does nothing for unauthenticated traffic. Most real systems layer limits:

• A coarse per-IP limit at the edge, before authentication, to absorb floods.
• A per-API-key or per-tenant limit after authentication, for fairness and quotas.
• A per-route limit on expensive endpoints, to protect specific dependencies.
• A global concurrency limit, to protect the process itself.

Each layer should have a distinct purpose and a distinct response. If the edge limit trips, the request never reaches your application, and the response is generic. If the per-tenant limit trips, the response should be specific and should include the quota headers the client needs to back off intelligently.

A subtle keying mistake is using a value the client controls without normalizing it. A limit keyed by a raw X-Forwarded-For header can be bypassed by spoofing the header if the proxy does not sanitize it. A limit keyed by an arbitrary tenant_id query parameter can be bypassed by changing the parameter. Limits must be keyed on values the server trusts.

Another subtlety: the key must be stable across the request path. If the edge keys on IP and the application keys on user ID, the two limits compose, but the client sees two different budgets and cannot reason about either. Document the composition.

The distributed race

A single-process token bucket is straightforward. A shared token bucket is a read-modify-write operation, and read-modify-write across processes is a concurrency problem.

The naive Redis version looks like this:

GET bucket:user:42          -> tokens = 1
# both requests see 1
SET bucket:user:42 0        -> both write 0
# both were allowed, but only one token was spent

This is the same lost update as a bank balance bug. The fix is to make the check-and-decrement atomic. The two standard approaches are:

1. A Lua script. Redis executes Lua atomically on the server, so the read, compute, and write happen as one operation.
2. A single atomic command. Redis 7's EXPIRE options, INCR with expiry, or a sorted-set-based sliding window can be made atomic without Lua, though complex algorithms usually need the script.

Here is the shape of a token bucket in Lua:

-- KEYS[1] = bucket key
-- ARGV[1] = capacity, ARGV[2] = refill rate per second,
-- ARGV[3] = now in milliseconds, ARGV[4] = cost
local capacity = tonumber(ARGV[1])
local rate = tonumber(ARGV[2])
local now = tonumber(ARGV[3])
local cost = tonumber(ARGV[4])

local data = redis.call("HMGET", KEYS[1], "tokens", "last")
local tokens = tonumber(data[1])
local last = tonumber(data[2])

if tokens == nil then
  tokens = capacity
  last = now
end

local elapsed = math.max(0, now - last) / 1000
tokens = math.min(capacity, tokens + elapsed * rate)

local allowed = tokens >= cost
if allowed then
  tokens = tokens - cost
end

redis.call("HSET", KEYS[1], "tokens", tokens, "last", now)
redis.call("PEXPIRE", KEYS[1], math.ceil((capacity / rate) * 2000))

return { allowed and 1 or 0, math.floor(tokens) }

The script is atomic, but atomicity is not the only distributed problem. The next few sections cover the ones that bite in production.

Clock skew

The Lua script above takes now from the application, not from Redis. That is a mistake unless every application instance has a tightly synchronized clock. Two instances with a 500 ms skew will disagree about how many tokens have refilled. One may over-refill and allow a burst; another may under-refill and reject valid requests.

The safer approach is to use Redis's own clock inside the script, via redis.call("TIME"). The key is that all participants must agree on one time source. If you must pass time from the application, use a monotonic source and monitor NTP offset, but understand that the limiter's accuracy is now bounded by clock skew.

For GCRA the problem is sharper: the stored TAT is a wall-clock timestamp, and a backward clock jump can make a bucket appear to be in the future indefinitely. Using Redis TIME avoids the multi-host skew, but a Redis failover to a replica with a different clock can still reset time. Some implementations store a monotonic offset instead.

The check-then-act problem across keys

A single Lua script makes one key's operation atomic. It does not make a multi-key decision atomic. If a request must satisfy a per-user limit and a per-tenant limit and a global limit, you have three independent checks. A request can pass the first two and fail the third, or pass all three but have another request interleave between them.

For most systems, this is acceptable: the limits are approximate protections, not accounting invariants. If you need strict multi-key atomicity, you can pass all keys into one Lua script and check them together in a deterministic order. That works in a single Redis instance but becomes much harder in a Redis Cluster, where keys may live on different shards and cross-slot scripts are not allowed.

The general pattern in a cluster is to make the keys hash to the same slot using hash tags, or to accept approximate composition. Again: decide whether the limit is a safety mechanism or a billing invariant. Safety mechanisms can be approximate. Billing invariants usually cannot.

Hot keys and the limiter as a bottleneck

A global limit is a single hot key. Every request in the system reads and writes it. At high throughput, that key becomes a serialization point: Redis processes commands on a single thread, so the global limiter can become the throughput ceiling for the entire service.

There are a few ways to deal with this:

• Shard the counter. Split the global budget across N sub-counters and let each request pick one at random. The sum approximates the global limit, and contention drops by a factor of N. The cost is accuracy: a shard can be exhausted while others have credit.
• Use local limiting plus periodic global reconciliation. Each instance allows a local rate and reports usage to a shared store at a coarse interval. The global limit is approximate, but the fast path is entirely in-process.
• Move the limit to the edge. A CDN or API gateway can absorb the hot key with specialized infrastructure and keep it out of your application's critical path.
• Use a different data store. Some systems use a dedicated rate-limiting service backed by sharded counters, or an in-memory store with a gossip protocol. The complexity is usually not worth it until the limiter is genuinely the bottleneck.

The meta-point: a limiter is a dependency on the request path. It must be faster and more available than the thing it protects, or it becomes the new single point of failure.

Fail open or fail closed?

If Redis is unavailable, what happens? This is a policy decision, and it should be explicit.

• Fail open allows requests when the limiter cannot make a decision. It preserves availability but removes protection exactly when the system may be under attack or overload. It is often right for availability-critical user-facing endpoints.
• Fail closed rejects requests when the limiter is unavailable. It preserves the protection but turns a limiter outage into a service outage. It is appropriate when the protected resource is more valuable than availability — a login endpoint under credential-stuffing attack, or a costly external API.

A hybrid is common: fail open to a conservative local limit. If the shared store is down, each instance enforces a small local budget derived from global_limit / instance_count. That keeps some protection without a hard outage. It is not exact, but it degrades in the right direction.

Whatever you choose, test it. A limiter that has never been exercised in its degraded mode will surprise you. Inject Redis timeouts in staging and verify the behavior, the logs, and the user-facing response.

What a good rejection looks like

A 429 is not just a status code. A correct rejection includes enough information for a well-behaved client to back off intelligently.

The essentials:

• Status 429 Too Many Requests. Not 403, not 503. 429 is the signal that the client should retry later, not that it is forbidden.
• Retry-After. Either a number of seconds or an HTTP date. This is the single most valuable header. Without it, clients guess, and guesses synchronize.
• Rate limit headers. The IETF draft defines RateLimit-Limit, RateLimit-Remaining, RateLimit-Reset, and a structured RateLimit field. Naming has changed over time, so many APIs still emit X-RateLimit-*. Pick one convention and document it.
• A machine-readable error body. Include the limit, the window, and the key or scope that was exceeded, without leaking sensitive identifiers.
• No side effects. Reject before doing the expensive work. A limiter that charges the client, writes a row, and then returns 429 has already done the damage.

function sendRateLimited(res, { retryAfterSeconds, limit, remaining, resetSeconds }) {
  res.setHeader("Retry-After", String(Math.max(1, Math.ceil(retryAfterSeconds))));
  res.setHeader("RateLimit-Limit", String(limit));
  res.setHeader("RateLimit-Remaining", String(Math.max(0, remaining)));
  res.setHeader("RateLimit-Reset", String(Math.max(0, Math.ceil(resetSeconds))));

  res.status(429).json({
    error: "rate_limited",
    message: "Too many requests. Retry after the indicated delay.",
    retryAfterSeconds: Math.max(1, Math.ceil(retryAfterSeconds)),
  });
}

Jitter is important here. If 10,000 clients receive the same Retry-After: 1, they will all retry at the same instant, creating a second spike. Add jitter, or cap the retry delay with a randomized component, or use a token that clients must poll. The general principle is that synchronized retries turn a short overload into a long one.

The same logic applies to exponential backoff. A client that retries at fixed intervals keeps offering the same load. A client that backs off exponentially with full jitter reduces its offered load over time, which gives the server room to recover.

Retry storms and the feedback loop

Rate limiting and retry policy are two halves of one system. A limiter without a retry contract is a source of retry storms. A retry policy without a limiter is a source of overload.

Consider what happens when a dependency slows down. Requests time out, clients retry, the dependency receives more requests than before, it slows further, and more requests time out. This is a metastable failure: the system was healthy at the same request rate a minute ago, and now it cannot recover even after the original trigger is gone.

The defenses are well known and frequently skipped:

• Retry budgets. Instead of retrying every failed request, allow retries up to a fraction of total traffic — say 10%. When the budget is exhausted, fail fast. This bounds the amplification factor.
• Circuit breakers. When a dependency's error rate exceeds a threshold, stop calling it for a cool-down period. This prevents the retry traffic from reaching a service that cannot handle it.
• Hedging with care. Sending a second request after a delay can reduce tail latency, but it doubles load on the dependency. Only hedge when the dependency has spare capacity, and cap the hedge rate.
• Idempotency keys. Retries are only safe if the operation can be repeated without duplicating side effects. A payment retry without an idempotency key is a double charge.
• Backoff with jitter. Fixed-delay retries synchronize; jittered exponential backoff de-synchronizes.

A useful framing is that rate limiting controls admission, while retries control amplification. The effective load on a dependency is not just the client request rate; it is the request rate times the average number of attempts. A limiter that allows 1,000 requests per second and a retry policy that averages three attempts has an effective limit of 3,000. Size the limit against the effective number, not the nominal one.

Multi-dimensional and cost-aware limiting

Real systems rarely have one limit. A request consumes several resources: CPU, database connections, external API quota, money. A single request-count limit is a proxy for all of them, and proxies are inaccurate.

A more precise design assigns a cost vector to each operation and enforces limits per dimension:

const operationCost = {
  "GET /search": { requests: 1, dbQueries: 3, externalCalls: 0 },
  "POST /checkout": { requests: 1, dbQueries: 12, externalCalls: 2 },
  "POST /report": { requests: 1, dbQueries: 50, externalCalls: 0 },
};

The request limiter may allow 1,000 requests per second, but the database limiter may only allow 2,000 queries per second. A burst of report generation would pass the request limit and exhaust the database budget, so the database dimension must be enforced separately — or the request must be rejected when its projected cost would exceed the remaining budget.

This is the essence of admission control: decide whether to accept work based on the resources it will consume, not just on a count. It is harder to implement than a counter, and it requires accurate cost estimates. But it is the difference between a limiter that protects the system and one that merely produces 429s while the database still melts.

The simplest useful version is a concurrency limit per dependency, combined with a weighted request limit. The concurrency limit bounds in-flight work, which is what actually exhausts connection pools and memory. The weighted limit bounds the long-run rate. Together they cover most of the failure modes.

Observability: you cannot tune what you cannot see

A rate limiter is a control system, and control systems need feedback. At minimum, record:

• Allowed and rejected counts, by key type and route. A rejection rate that jumps is either an attack, a misconfigured client, or a limit that is too tight.
• Rejection reasons, including which dimension tripped.
• Limiter latency, including the shared-store round trip. If the limiter adds 20 ms to every request, it is a performance problem regardless of its accuracy.
• Shared-store health, including error rate, timeout rate, and fail-open/fail-closed activations.
• Top keys by consumption. This is how you find the noisy neighbor, the leaking API key, and the scraper.
• Retry-after distribution. If most rejections ask for one second, the limit is probably too tight or the clients are too aggressive.

A dashboard that only shows total 429s is not enough. You need to know who is being limited and why, or you will tune blindly.

Testing a limiter

Rate limiters are concurrency code, and concurrency code needs adversarial tests. A reasonable test suite includes:

1. Deterministic algorithm tests with a fake clock, like the examples in this article. Verify the burst, the refill, the retry-after calculation, and the boundary behavior.
2. Concurrency tests against the shared store. Fire N simultaneous requests with a limit of K and assert that exactly K succeed. Run this repeatedly; race conditions are probabilistic.
3. Clock tests. Move the clock backward and forward. Verify that the limiter does not create tokens or lock out a key forever.
4. Failure injection. Make the store time out, return errors, and disappear. Verify the configured fail-open or fail-closed behavior, and that the metric is recorded.
5. Load tests. Measure the limiter's own latency and throughput at the target request rate. A limiter that adds a synchronous network round trip to every request may be unacceptable for a low-latency API.
6. Boundary tests for the window. Confirm that the chosen algorithm does not allow the overshoot you are trying to prevent.

The concurrency test is the one people skip, and it is the one that catches the bug that matters. A Lua script is atomic, but the code that calls it, parses the result, and decides what to do is not. If the decision logic lives in the application, the atomicity of the store does not save you.

A design checklist

When you add a rate limiter, answer these questions before writing code:

• What resource is being protected, and what is its actual capacity? Measure, do not guess.
• What is the failure mode if the limit is wrong? Too tight causes outages for legitimate users; too loose fails to protect the resource.
• What is the key, and can the client forge it? Use server-trusted identifiers.
• Rate, concurrency, or quota — or all three? They protect different things.
• What algorithm matches the burst profile? Token bucket for bursty traffic, GCRA for smooth output, sliding log for accuracy on low-volume operations, fixed windows only for calendar-based quotas.
• Where does the state live, and how is it made atomic? Lua, atomic commands, or a single-writer design.
• What happens when the store is unavailable? Fail open, fail closed, or degrade to a local limit. Decide explicitly.
• What does the client see on rejection? 429, Retry-After, quota headers, and a documented backoff policy.
• How do retries interact with the limit? Budget them, jitter them, and cap the amplification.
• How will you observe and tune it? Instrument allowances, rejections, latency, and top keys.
• How will you test the race? Concurrency tests against the real store, not just unit tests with a fake.

The short version

A rate limiter is not a counter. It is a small distributed system that sits on the request path, and it inherits every distributed-systems problem: races, clocks, hot keys, partial failure, and the tension between availability and protection.

The algorithms are well understood. A token bucket gives you burst control and a sustainable rate with two numbers of state. GCRA gives you the same behavior with one timestamp. Sliding windows trade exactness for memory. Fixed windows are cheap and wrong at the boundary. The hard parts are key design, atomicity in the shared store, and deciding what happens when the limiter itself fails.

Get those right, and the limiter becomes a quiet safety mechanism. Get them wrong, and it becomes a source of outages that only appear under the load it was meant to handle.