Latency vs Throughput in System Design

A customer waits for an order confirmation. The time from sending the request to receiving the response is latency. During that same second, the service may complete many other customers’ requests. The rate of completed work is throughput.

A system can finish a lot of work while some people wait too long. It can also respond quickly to a few requests but struggle when many arrive together. A useful design states both the waiting-time goal and the amount of work it must handle.

Latency needs a boundary. Browser-to-browser latency may include network travel, server queues and backend work. A database’s query timer measures a smaller part. Two dashboards can show different numbers because they measure different boundaries.

Throughput also needs a window and unit: for example, 500 successful requests completed in 10 seconds is 50 successful requests per second. Arrivals are demand, not necessarily completed throughput. Count errors separately so fast failures do not look like successful performance.

Time inside a worker is service time. Waiting for an available worker is queue time. A fast operation can still have high total latency when it waits behind many other operations.

Eight requests arrive together at time zero. Each worker handles one request at a time. Change the worker count and the service time to see the finish time of every request. More workers reduce the queue in this model; they do not shorten the work inside each request.

The lab assumes independent identical workers, constant service time and no network, startup or shared-resource overhead. All requests succeed. It is a deterministic teaching model, not a live performance test.

The batch throughput counts eight completions divided by the time until the final completion. The ideal steady capacity uses workers ÷ service time. These can differ when the final wave does not fill every worker. The p50 and p95 use the nearest-rank method on only eight finish times; a tiny batch is not evidence of production tail latency.

Imagine nine requests each take 50 ms and one takes 1,000 ms. The average is 145 ms. Most customers got a much faster answer, while one waited a full second. One average cannot describe both experiences.

p50, the median, describes the middle of an ordered sample. p95 describes a latency threshold at or below which about 95% of observations fall, subject to the calculation method. Larger percentiles focus on slower responses. They are not the maximum or a promise for every future request.

Collect enough observations over a stated interval, separate request types and inspect errors. Do not average percentile values from separate servers to obtain the fleet percentile; use an appropriate combined distribution. Google’s monitoring workbook explains why latency distributions matter.

Parallel workers help only when work can proceed independently and resources remain available. If all workers compete for one database connection or a busy disk, increasing their number may mostly create more waiting.

Look at CPU, memory, network, storage and dependency queues alongside latency. Follow a request across its calls to find where time is spent. A long queue can indicate demand exceeds the useful completion rate; accepting unlimited work can increase latency until requests time out.

Queues can smooth short bursts, but do not create processing capacity. Use bounded queues, appropriate limits and a recovery plan. A client retry can add even more load, so retry behaviour belongs in the experiment.

A checkout is slow at low traffic

Investigate its actual request path: expensive queries, serial network calls or large responses. More workers may increase concurrent capacity without shortening that single checkout. Measure before deciding which step to optimise.

A file-processing backlog grows

If tasks are independent and storage can keep up, more workers may increase useful throughput. Track job completion time and the oldest queued job as well as the processing rate. A large completion count can still hide a customer waiting for hours.

A launch creates a short traffic spike

Test pre-warmed capacity, admission limits and bounded waiting against the user’s deadline. Batching can improve efficiency but adds waiting before work starts. Caching can avoid suitable repeated work, with freshness and correctness trade-offs.

k6 load-test guidance helps structure representative normal-load and spike experiments. Record successful throughput, latency percentiles and errors together. Official sources checked 6 October 2026.

Latency describes waiting for a particular request; throughput describes completed work per unit of time. State the boundaries and units. Queue time and service time contribute differently to the experience.

Use parallelism when work and dependencies support it. Inspect slow responses, errors and resource limits alongside throughput. The best improvement is the one that meets the user’s goal under realistic demand.