Catalog
concept#Integration#Platform#Architecture#Reliability

Message Queue

A structural integration pattern for asynchronous delivery of messages between system components.

Message queues are a structural integration pattern that enable asynchronous communication between system components by temporarily storing and reliably delivering messages.
Established
Medium

Classification

  • Medium
  • Technical
  • Architectural
  • Intermediate

Technical context

HTTP-based producer/consumer servicesDatabases for persistence and event sourcingMonitoring and observability tools (e.g., Prometheus)

Principles & goals

Decouple components to reduce direct dependencies.Idempotent processing to safely repeat message handling.Define explicit error paths and dead-letter mechanisms.
Build
Domain, Team

Use cases & scenarios

Compromises

  • Missing backpressure control can lead to resource exhaustion.
  • Inconsistent message processing if idempotency is missing.
  • Data loss with incorrect persistence or replication configuration.
  • Implement idempotent consumers.
  • Set up SLA-oriented monitoring of queue length and latency.
  • Define dead-letter queues and clear error-handling processes.

I/O & resources

  • Producing applications or services
  • Message format and schema definitions
  • Configured messaging broker and network access
  • Reliably delivered messages to consumers
  • Metrics for throughput, latency and errors
  • Dead-letter and error logs for post-processing

Description

Message queues are a structural integration pattern that enable asynchronous communication between system components by temporarily storing and reliably delivering messages. They decouple producers and consumers, support load leveling, fault isolation, and retry strategies. Common use cases include microservices, event-driven architectures, background processing, and buffering for peak loads. Architectural choices affect durability, throughput, and latency.

✔

  • Improved scalability via asynchronous buffering.
  • Increased fault tolerance via decoupling and retries.
  • Smoothing of peak loads without directly taxing consumers.

✖

  • Additional complexity in operation and monitoring.
  • Potentially increased end-to-end latency due to asynchronous processing.
  • Requires clear conventions for message formats and error handling.

  • Throughput (messages/s)

    Number of successfully delivered messages per second.

  • Queue length

    Current number of unprocessed messages in the queue.

  • Time to delivery (end-to-end latency)

    Average time from message creation to successful processing.

RabbitMQ for robust work processing

RabbitMQ used as broker for reliable delivery and dead-letter handling in a microservices system.

Amazon SQS for elastic queuing models

SQS as fully managed service to buffer peak loads and enable simple scaling.

Kafka for high-throughput event streams

Apache Kafka is used as distributed log when ordering, high throughput and replication are central.

1

Define requirements and throughput/latency goals.

2

Select broker type (e.g., AMQP, Kafka, SQS) and prototype test.

3

Define and integrate message formats, retry and dead-letter strategies.

⚠️ Technical debt & bottlenecks

  • Monolithic queue configuration without partitioning or prioritization.
  • Missing schema registry leads to inconsistent message formats.
  • Insufficient replication and backup strategies for persistence.
Network bandwidthBroker CPU/IOStorage/persistence limits
  • Using a queue as primary database without replication guarantees.
  • Persisting all messages indefinitely without retention strategy.
  • Missing idempotency causing duplicate order processing.
  • Underestimating operationalization (backup, monitoring, scaling).
  • Wrong assumptions about message ordering in distributed setups.
  • Untested failure scenarios for broker partitions.
Understanding of distributed systems and messaging principlesOperation and tuning of message brokersError handling, idempotency and schema management
Throughput requirementsLatency objectivesAvailability and durability requirements
  • • Throughput limits of the chosen broker.
  • • Compliance requirements for data retention and encryption.
  • • Network latency between producers, broker and consumers.