Understanding Load Balancers in System Design

Imagine that an online shop becomes so popular that one computer cannot answer every visitor quickly enough. The team adds two more computers to share the work. But visitors still need one address to open the shop, and someone must decide which computer handles each request.

A server is a computer that receives a request and sends a response. A load balancer sits in front of several servers and sends incoming work to an available one. It helps the servers work together, but it cannot make each server do more work than it can handle.

When you open the shop, your browser sends a request—for example, “show me the products.” The load balancer receives that request and chooses a server to answer it. The response then makes its way back to your browser.

One simple choice is round robin: send work to server A, then B, then C, and repeat. Another choice, least connections, prefers a server with fewer open connections. A connection is a communication channel between computers; fewer connections do not always mean less work, because some requests take longer than others.

The balancer regularly checks whether servers respond. If a server stops responding, it can stop sending new work there after detecting the problem. A request already on its way may still fail. Information such as a shopping basket should be accessible to whichever server handles the next request.

Suppose each server can answer 500 requests in one second. Three servers can therefore answer up to 1,500 requests per second in this simplified example. This is their capacity: the amount of work they can handle in the time available.

Move the incoming-requests slider to change how much work arrives. Then select “Make one server unavailable”. A backend is simply a server behind the load balancer. Watch whether the remaining servers can keep up. “Pending growth” shows how much unfinished work could accumulate each second if excess requests are allowed to wait.

Try starting at 1,200 requests per second with three servers, then fail one. Can you explain why the remaining capacity is too small?

You do not need to buy an appliance to try load balancing. Some tools are free software that you run on your own computer or server. Others are paid software, cloud services or physical devices. The choice changes who installs, updates and looks after the load balancer.

Free software does not mean free hosting. Running a tool on a rented server still costs money. You also spend time on configuration, updates and monitoring. A paid product can include support or take over some of that work.

A practical starting choice: use HAProxy Community to learn the basic flow locally. If your application later lives in AWS and you want AWS to operate the load balancer, evaluate ALB. If you need vendor support on your own servers, compare the commercial software options. Your requirements decide the choice; a paid label does not by itself imply higher throughput.

These are representative choices, not every product available. Licensing and billing models were checked on 6 October 2026; follow the product links for current terms.

Verified 6 October 2026. These are representative implementations, not an exhaustive inventory. Product throughput depends on payloads, TLS, connection reuse, CPU, network and configuration.

Requests per second measures how much work gets done. Response time measures how long a visitor waits. A p99 response time describes a threshold that 99% of measured responses meet; the slowest 1% take longer. TLS is the encryption used to protect traffic, and setting up encrypted connections also uses resources.

• HAProxy 3.4 documentation: software proxy with round-robin and least-connections policies. Benchmark requests/s, concurrent connections and p99 latency on your chosen host.
• Envoy: programmable proxy and load-balancing policies. Its latest documentation may include development features; verify the deployed stable release.
• AWS Application Load Balancer: managed HTTP routing. Quotas and capacity units describe different dimensions; they are not a universal requests/s guarantee.

A hardware appliance comparison must include licensed throughput, TLS transactions/s, interfaces, packet sizes and failover mode. A 10 Gbit/s interface has a raw upper bound of 1.25 GB/s before protocol overhead; application throughput can be much lower. The lab assumes 500 requests/s per backend for teaching only, not any product benchmark.

Production means the system is serving real users. Let’s look more closely at three common choices: HAProxy, NGINX and AWS Application Load Balancer. They represent a dedicated software load balancer, a web server that also balances traffic, and a managed cloud service. This is a practical shortlist rather than a ranking by market share.

HAProxy: a dedicated traffic distributor

Imagine several application servers answering the same API requests. An API is a way for one program to ask another for information or an action. HAProxy sits in front of those servers and decides which healthy server should receive each request. It supports both HTTP web traffic and TCP connections, which can carry other kinds of application traffic. It offers balancing policies and active health checks that probe servers even when visitors are not sending requests.

What you take on: with the free Community version, your team runs the hosts, updates the software, configures checks and arranges another load balancer for failover. It is a useful candidate when traffic routing is the main job and you want control over it. Official HAProxy overview · Configuration reference.

NGINX: web serving and load balancing together

A website may need to serve images and stylesheets as well as forward requests to application servers. NGINX can do both. Its HTTP load balancer supports round robin and least connections, so it can share application requests while also acting as the website’s front door.

What you take on: you still operate the hosts and configure recovery. In NGINX Open Source, passive checks can mark a server unavailable after configured request failures. Periodic active HTTP health checks are a commercial feature; do not assume that Open Source and NGINX Plus offer identical capabilities. If your team already runs NGINX, extending that setup may be simpler than introducing another tool. HTTP balancing guide · Server groups and failure handling.

AWS ALB: AWS operates the load-balancing service

Application Load Balancer, usually shortened to ALB, routes HTTP and HTTPS requests to application targets in AWS. A target is a destination that can answer a request. Rules can send /images and /orders to different groups of servers. You configure targets, health checks and routing; AWS operates the load-balancing infrastructure.

What you take on: you pay running and usage charges, work within AWS service limits, and still keep your application servers healthy. Spread the application across Availability Zones—separate locations within an AWS region—to reduce exposure to a location failure. ALB removes host maintenance for the balancer, but it does not automatically fix the application. ALB features · Deployment guide · Pricing.

Capabilities checked on 6 October 2026. Next, use requirements to decide which of these approaches fits a particular system.

There is no winner for every system. Start with where the application runs, what traffic it receives, who will maintain it, and what happens during a failure. The choices below are reasoned starting points for the stated requirements, not measured performance rankings.

1. A small website already uses NGINX

Requirements: two rented application servers, website images and stylesheets, a small budget, and a team comfortable maintaining NGINX.

Start with NGINX Open Source. It can serve files and balance HTTP requests in the setup the team already understands. Adding HAProxy might create another system to maintain without meeting a new need. Trade-off: the team owns updates and failover, and must check whether passive failure detection meets its needs. If periodic active HTTP checks or vendor support are required, compare NGINX Plus with HAProxy instead.

2. A team needs dedicated routing on its own servers

Requirements: servers in the team’s own data centre, both web requests and TCP connections, active checks, and staff who can operate redundant load balancers.

Start with HAProxy. Its dedicated routing role and HTTP/TCP support fit those needs without requiring a cloud move. NGINX may also fit some workloads, but serving website files is not part of this requirement. Trade-off: HAProxy Community avoids a software licence fee while leaving configuration, monitoring and recovery with the team. Compare commercial support if staff need help operating it.

3. An AWS web application has a small operations team

Requirements: HTTP/HTTPS traffic, separate routes for catalogue and checkout, application servers across two Availability Zones, and little time to maintain load-balancer hosts.

Start with AWS ALB. Its request-routing rules and managed infrastructure match the hosting environment and reduce host maintenance. Self-hosted HAProxy or NGINX may offer more configuration control, but the team would have to operate them. Trade-off: ALB adds ongoing usage charges and AWS-specific configuration. Compare the full operating cost, including staff time; a free software licence alone does not settle the decision.

4. The traffic is not HTTP

Requirements: a custom application uses long-lived TCP connections rather than website requests.

ALB is not the right fit for this traffic. Evaluate HAProxy’s TCP mode if the team wants to run its own load balancers. In AWS, also evaluate Network Load Balancer (NLB), a different service that handles connection-level traffic. Trade-off: connection routing does not provide ALB’s HTTP path rules. Check protocol support, connection duration, address requirements and operating cost before choosing.

Check capacity before accepting any choice

Suppose peak traffic is 2,500 requests per second. In a teaching example, each application server sustains 1,500 requests per second at an acceptable response time. Two servers handle the peak together, but only 1,500 requests per second remain if one fails. Three servers leave 3,000 requests per second after one failure, provided their shared database and other dependencies can keep up. These numbers are assumptions, not product specifications.

Choosing a different load-balancer brand does not repair that capacity gap. Test your chosen balancer and application together with realistic request sizes, encrypted connections, latency targets and a failed server. Revisit the choice if the measurements or the team’s operating needs change.

This optional exercise runs one HAProxy load balancer and two tiny web servers. Each server returns its own name, so you can see which one answered. You do not need a cloud account, and you can complete the lesson without running this exercise.

You need a running Docker installation with Docker Compose and an unused local port 8080. Docker runs the three programs in separate containers, which are isolated processes that can communicate on a shared network. The first start downloads their images from the internet. HAProxy is free software; Docker Desktop has separate licensing terms.

Download the complete local lab (.zip). Unzip it and open a terminal in the load-balancer-lab folder. The included README contains the full sequence and cleanup command.

The folder contains compose.yaml, haproxy.cfg and two folders named server-a and server-b. Each server folder contains an index.html file that says either Server A or Server B.

services:
  balancer:
    image: haproxy:3.4.6
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
    depends_on:
      - app_a
      - app_b
  app_a:
    image: python:3.13.16-alpine
    command: ["python", "-m", "http.server", "8000", "--directory", "/site"]
    volumes:
      - ./server-a:/site:ro
  app_b:
    image: python:3.13.16-alpine
    command: ["python", "-m", "http.server", "8000", "--directory", "/site"]
    volumes:
      - ./server-b:/site:ro

HAProxy reads this configuration. The frontend accepts your requests at port 8080. The backend lists the servers that can answer. roundrobin shares new requests between healthy servers in turn.

defaults
  mode http
  timeout connect 5s
  timeout client 30s
  timeout server 30s

frontend local_shop
  bind :8080
  default_backend shop_servers

backend shop_servers
  balance roundrobin
  server a app_a:8000 check inter 1s fall 2 rise 2
  server b app_b:8000 check inter 1s fall 2 rise 2

Validate the Compose file, then start all three programs:

docker compose config
docker compose up -d

Now check that HAProxy accepts its configuration:

docker compose exec balancer haproxy -c -f /usr/local/etc/haproxy/haproxy.cfg

If startup fails, use docker compose logs to see the messages. If port 8080 is occupied, change the host port in the Compose file to 127.0.0.1:8081:8080 and use 8081 in the requests below.

With the programs running, send a request through HAProxy:

curl http://127.0.0.1:8080/

Run that command several times. You should see Server A and Server B take turns responding while both are healthy. Each new curl command creates a new connection, making the round-robin behaviour easier to observe than repeated browser refreshes.

Now predict the result: if Server B stops, should visitors still receive an answer? Stop it and wait about three seconds for the health checks to notice:

docker compose stop app_b

Repeat the curl command. After failure detection, only Server A should answer. A request during the transition can fail. Here, HAProxy checks once per second and marks a server unavailable after two failed checks. These checks test whether a connection opens; they do not prove that an application can complete an order.

Bring Server B back, wait a few seconds, and repeat the requests:

docker compose start app_b

Once it passes two checks, both servers should answer again. You have now seen real request routing and failure detection, rather than just a capacity simulation.

Finish by stopping and removing the exercise containers:

docker compose down

This small exercise does not measure maximum throughput. It also uses one load balancer, so the load balancer itself is still a single point of failure. A production design needs to consider that failure too.

A layer-4 balancer chooses a destination using connection information such as addresses and ports. A layer-7 balancer understands an application protocol such as HTTP and can route using a hostname or path. These are different capabilities; neither layer is automatically best for every workload.

For a shop, /images and /checkout might need different HTTP backends. For a generic TCP service, connection routing may be enough. Decide where TLS terminates and whether the next hop also needs encryption. Connections, requests, bytes, handshake cost and routing rules all affect capacity.

If a trusted proxy adds client-address headers, the application must accept them only under a deliberate trusted-proxy policy. Do not treat arbitrary client-supplied forwarding headers as authenticated identity. See protocols and proxies for the connection boundary.

A load balancer gives visitors one entrance and shares their requests among available servers. It can avoid a failed server after detecting the failure, but it cannot replace that server’s lost capacity.

Leave enough spare capacity for a busy period or a server failure. When evaluating a real system, measure both how many requests it handles and how long visitors wait for a response. The balancer itself also needs a plan for failure.