Message Queues (e.g., Kafka, RabbitMQ)

A shop accepts an order, then needs to send a confirmation email. The visitor should not have to wait while the email provider is slow. A message queue can hold a description of that work until a background worker is ready.

The producer creates a message. The queue holds it according to its configured guarantees. A consumer, or worker, receives it and performs the task. Accepting a message does not mean the task has finished, so the app must communicate the right status to its user.

An acknowledgement tells the queue that a delivery can be considered handled. For reliable work, the application normally acknowledges after the intended result has been safely recorded. If a worker disappears before acknowledging, the message may become available for another attempt.

That protects against lost work, but it introduces duplicates: the first attempt may have saved its result before the acknowledgement was lost. Design an idempotent handler that can recognise the same intended job and avoid repeating its effect. A durable job ID and result record may help; external actions need their own duplicate-prevention strategy.

A publisher confirmation covers the producer-to-broker step. A worker acknowledgement covers a different step. Neither should be mistaken for an end-to-end guarantee that an email was delivered exactly once.

Deliver J1, perform its work, and simulate a crash before acknowledging it. Deliver it again and perform the work again. With duplicate protection enabled, the example recognises the job ID and skips its already-recorded effect.

Reset and turn protection off to see a duplicate effect. This model lives only in browser memory; a production deduplication record needs appropriate persistence and coordination. A real external email provider may need an idempotency mechanism of its own.

Suppose arrivals average 120 jobs per second and workers finish 100 jobs per second. Under these teaching assumptions, pending work grows by 20 jobs every second. A queue stores that gap; it does not remove it. Adding consumers helps only if the downstream service can handle the extra work.

Monitor queue depth and the age of the oldest pending job. A small queue can still contain an old stuck job. Limit outstanding work, slow producers where possible and define what the app does when capacity is exhausted. This control of incoming work is often called backpressure.

Retry temporary failures with delays and a maximum attempt count. Repeatedly failing jobs can move to a dead-letter queue for investigation instead of blocking normal work indefinitely. Keep the reason and job ID, and plan a safe way to inspect or retry those jobs.

Representative official sources checked 6 October 2026. Compare message size, sustained arrival/completion rates, latency, retention, duplicates, ordering and failure recovery. Published throughput depends on configuration and workload.

RabbitMQ: routing and background work

RabbitMQ is open-source messaging software with routing and consumer delivery controls. Evaluate it for task distribution and application messaging when your team can operate the broker or choose a managed provider. Its publisher confirms and consumer acknowledgements solve different steps of the delivery path.

Apache Kafka: retained events and independent readers

Apache Kafka is open-source event-streaming software. Evaluate its partitioned, retained log when several applications need to read or replay events independently. Ordering is scoped to a partition under the relevant producer/consumer configuration, not automatically to every event in a system. Retention, partitions and consumer progress add operational decisions.

Amazon SQS: managed queueing in AWS

SQS removes broker-host maintenance and has usage-based charges. Standard queues can redeliver messages; choose queue type and application handling from the required ordering and delivery semantics. Queue-level deduplication does not make every external side effect exactly once.

For a confirmation-email task, start with a work-queue design. For replayable events feeding several independent applications, evaluate a retained log. Free software still needs hosting, monitoring and recovery; managed operation still needs correct consumer logic.

A work queue normally shares jobs among competing workers in the same processing group. In publish/subscribe, separate subscriptions can each receive a copy of the published fact. Adding a second worker to one group is different from adding a second independent consumer such as analytics.

A retained log can let consumer groups maintain their own positions and replay available history. Retention still has a configured boundary; it is not indefinite free storage. Ordering is often per queue or partition, not automatically global across all keys.

A repeatedly failing message needs bounded retries, a diagnostic dead-letter or quarantine path, and a repair/replay policy. Preserve identity when replaying and protect sensitive payloads. Silent discard and endless immediate retries both hide useful failure information. Continue with stream processing for stateful aggregation.

Backpressure means slowing or rejecting upstream work when downstream capacity cannot keep up. Bound worker concurrency and queue depth, monitor the age of the oldest job and define what admission does when the budget is exhausted. An ever-growing queue only moves the wait out of the request path.

Priority queues can help urgent work but may starve ordinary jobs unless the policy preserves fairness. For large payloads, a message can carry a protected object reference instead of every byte; this is sometimes called the claim-check pattern. Keep reference expiry, permissions and object cleanup consistent with retries.

An asynchronous request can return an accepted status and a job identity. Give the client a status endpoint or a supported callback, and distinguish pending, completed and failed outcomes. Acceptance is not completion; the client needs a deadline and a way to resolve an uncertain response.

Use Publisher/Subscriber for independent subscriptions, Outbox for committed publication intent, and Asynchronous Request/Reply for client-visible job outcomes. Delivery, source commit and business completion remain separate promises.

A queue helps move suitable work out of the visitor’s waiting path and absorb short bursts. It needs an explicit delivery, acknowledgement and retry design.

Use stable job identities, make repeated attempts safe, monitor waiting time and capacity, and handle persistent failures deliberately. Choose a task queue or retained event log from how the work must be consumed.