Message Queues & Event Streaming

Intermediate · 20 min read · High-Level Design, Services & Communication

Queues vs logs, delivery guarantees, ordering, consumer groups and back-pressure.

So far, communication has been synchronous: a caller sends a request and waits for the response. That couples the caller to the callee's availability and speed. Asynchronous messaging inserts a durable intermediary — a queue or log — so producers hand off work and move on, and consumers process it at their own pace.

Message queues and event streams are among the most frequently used building blocks in HLD interviews.

Why go asynchronous

In the synchronous version, the order API's latency is the sum of all downstream calls, and if the email service is down, placing an order fails. The asynchronous version fixes both:

Two models: queues and logs

Messages are delivered to one consumer among a pool of competing consumers, and are deleted once acknowledged.

How Kafka achieves scale and ordering

Delivery semantics

Failures can happen at any point: a producer may not receive the broker's ack, or a consumer may crash after processing but before acknowledging.

Handling failures

Back-pressure and consumer lag

If producers outpace consumers, the backlog grows. Consumer lag (how far behind the latest offset a consumer group is) is the key metric to monitor and to auto-scale on.

The dual-write problem and the outbox pattern

A service often needs to update its database and publish an event — e.g., save an order and emit OrderCreated. Doing these as two separate steps is unsafe:

Common use cases in designs

Interview questions

  • How do you guarantee a message is processed exactly once?
  • You need all events for the same order processed in order, but you also need high throughput. How?
  • When would you use RabbitMQ/SQS instead of Kafka?
  • What is a poison message and how do you handle it?

Practice

Write your answer before opening the suggested one. Anything you mark "Partly" or "Missed it" comes back in your review deck.

Further reading

Key takeaways

  • Async messaging decouples services, absorbs spikes, and moves slow work off the request path — at the cost of eventual consistency.
  • Queues distribute tasks to competing workers and delete messages once acknowledged; logs retain ordered, replayable events for many consumer groups.
  • Kafka orders messages per partition; use the entity ID as the key, and remember that partitions cap consumer parallelism.
  • Design for at-least-once delivery with idempotent consumers; use retries, backoff and dead-letter queues.
  • Use the transactional outbox (or CDC) to update a database and publish events reliably.

Previous: Real-Time Communication · Next: Monoliths, Microservices & Service Discovery