Rate Limiting in System Design

An API may receive a sudden burst from one client while other people are trying to use it. A rate limiter controls how much work is accepted over time. It can protect capacity and improve fairness, but it must be designed around the application’s real limits.

The policy needs a key, such as a signed-in account, a budget and a response when the budget is exhausted. A client limit is different from a global capacity limit. Many individually compliant clients can still overload a shared service.

A fixed window counts requests during a period and resets at its boundary. It is simple, but requests just before and just after the boundary can create a larger short burst than expected. A rolling-window policy considers a moving period, with additional state or approximation costs.

A token bucket holds a limited number of permits. A request spends a token; tokens refill over time up to a maximum. The refill rate controls longer-term admission and the bucket size allows a bounded burst. A full bucket does not mean the service can safely process that entire burst at once.

A concurrency limit caps work currently in progress, rather than requests over time. Slow requests can exhaust concurrency even when the request rate looks modest. These controls answer different questions and may need to work together.

Each demo client starts with three tokens. A request consumes one token; an empty bucket returns 429. Advancing one simulated second adds one token to each client, capped at three. No timer runs automatically.

Send a burst of four requests, then refill and try again. Switch clients to see separate budgets. These capacities are teaching choices, not production limits. Giving both clients their own budgets does not impose one global service budget.

Account-based limits can separate known customers, while IP-based limits may group unrelated people behind one shared network. Attackers may change addresses or use many accounts. A key is therefore a policy choice, not a complete abuse-prevention system. A more expensive operation may also need a different budget from a cheap read.

For an HTTP rate-limit rejection, 429 Too Many Requests tells the client to slow down. Where appropriate, a Retry-After value explains when to try again. Clients should use a delay and spread retries rather than all retrying immediately together. Repeating a write still requires a safe application-level retry design.

Track accepted and rejected work, latency and unfair throttling reports. A limiter that protects servers while rejecting normal customers needs its policy reviewed.

If each of three gateway instances independently allows 100 requests/s for the same account, that account may get roughly three times the intended shared allowance. Central state or another coordination strategy can enforce a shared policy, but it adds network calls, cost and a dependency.

Updates to a shared counter or bucket must be coordinated atomically so simultaneous requests cannot all spend the same allowance. Local budgets can reduce coordination work, with explicit accuracy and burst trade-offs. Do not assume that any distributed rate limiter is an exact global ceiling.

Decide what happens when the limiter cannot reach its state store. Fail open accepts work and risks overload or abuse. Fail closed rejects work and risks blocking legitimate use. The decision can differ for a public catalogue and a sensitive write. Measure failure behaviour as well as steady traffic.

Representative sources checked 6 October 2026. Capacity depends on request mix, key count, storage/coordination and infrastructure; benchmark the configured request path rather than assigning a universal requests/s value.

NGINX on your existing hosts

The open-source limit_req module uses a leaky-bucket method with configurable burst behaviour. Evaluate it for a straightforward edge policy in an existing deployment. Its algorithm is not the token-bucket browser demo, and independent hosts need a deliberate shared-limit strategy.

Envoy with a rate-limit service

Envoy’s rate-limit filter can consult a separate service. This can centralise policy for several proxies while adding a dependency and operational work. Check the stable version you deploy; latest documentation may describe development features.

Amazon API Gateway in AWS

The managed service provides token-bucket throttling targets. AWS describes throttling as best effort rather than an exact request ceiling. API types have different policy features; check them and usage costs.

Use an existing proxy for a simple local edge limit, coordinated policy when several servers must share a customer budget, or managed gateway controls when that fits your hosting and feature requirements. Application-level checks remain useful for expensive operations and business-specific quotas.

Choose the key, sustained allowance, burst allowance and rejection behaviour deliberately. Rate, concurrency and global capacity are different controls.

Distributed coordination and state-store failure affect the promise you can make. Measure the complete path, explain rejections and make client retries safe.