Design a Distributed Rate Limiter

Intermediate · 22 min read · High-Level Design, Case Studies: Core

A rate-limiting service shared by many API servers.

Design a rate limiter used by an API platform: hundreds of API gateway instances must enforce limits like "1,000 requests per minute per API key" and "10 login attempts per hour per IP", consistently, while adding almost no latency.

1. Clarify the problem

2. Requirements

Functional

3. Capacity estimation

So: state fits easily in memory; throughput, not size, drives the shard count. A sharded Redis cluster is a natural fit.

4. API

Internal check API (called by the gateway, or embedded as a library):

5. Data model

Rules — small, read-mostly configuration: stored in a database, cached in every gateway's memory, refreshed via a watch/pub-sub or every 30 s.

6. High-level design

Request flow

7. Deep dives

Token bucket for API quotas (allows natural bursts, two numbers of state) and sliding window counter where precise per-window limits are contractual. Both run as a single Lua script per counter — atomic on the Redis server, so concurrent gateways can't double-spend tokens.

8. Observability

9. Trade-offs summary

10. What interviewers look for

Interview questions

  • Two gateway instances check the same key at the same moment. How do you avoid both allowing the last request?
  • How would you rate-limit a client to 10 requests per second across three regions without adding cross-region latency?
  • What headers should a rate-limited API return, and why?

Practice

Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.

Further reading

Key takeaways

  • Enforce in the gateway with an in-memory rule cache and a shared, sharded counter store.
  • Use atomic server-side scripts (Lua with Redis TIME) to avoid races and clock skew.
  • Cut latency and load with pipelining, local deny caching and token leasing for hot keys.
  • Choose fail-open or fail-closed per rule, with tight timeouts and circuit breakers.
  • Across regions, split budgets and synchronise asynchronously.

Previous: Design a URL Shortener · Next: Design a Notification System