Catalog
concept#Integration#Architecture#Observability#Reliability

Messaging Queues

Mechanism for asynchronous communication between components by persisting and delivering messages in a decoupled, ordered manner.

Messaging queues are architectural components that decouple producers and consumers by persisting messages for asynchronous delivery.
Established
Medium

Classification

  • Medium
  • Technical
  • Architectural
  • Intermediate

Technical context

Message brokers (e.g., RabbitMQ, Kafka, SQS)Consumer services/workersMonitoring and observability stacks

Principles & goals

Decouple producers and consumersDefine explicit delivery guaranteesImplement monitoring and backpressure
Build
Domain, Team

Use cases & scenarios

Compromises

  • Message loss due to misconfiguration
  • Queue overload leads to increasing latency
  • Incorrect delivery semantics cause inconsistencies
  • Implement idempotent consumers
  • Configure dead-letter queues and retries
  • Set up metrics and alerts for queue length and latency

I/O & resources

  • Produced messages/events
  • Message formats and schemas
  • Authentication and authorization data
  • Processed messages by consumers
  • Monitoring and metrics for observability
  • Error and dead-letter entries

Description

Messaging queues are architectural components that decouple producers and consumers by persisting messages for asynchronous delivery. They enable load leveling, resilience to failures, and scalable event-driven integrations across services. Typical concerns include delivery semantics, ordering guarantees, capacity management, and operational complexity.

✔

  • Increased resilience through asynchronous processing
  • Improved load distribution and scalability
  • Decoupling enables independent development

✖

  • Additional infrastructure and operational overhead
  • Latency due to persistence and queue processing
  • Complexity around ordering and transactional guarantees

  • Queue length

    Number of unprocessed messages per queue; indicator of backlog.

  • Throughput (messages/s)

    Number of processed messages per second; measures capacity.

  • Delivery times / latency

    Time from creation to processing; important for SLAs.

RabbitMQ in microservice architecture

Use of RabbitMQ for reliable task distribution and command synchronization between services.

Kafka for event streaming

Apache Kafka as a distributed log-based platform for high throughput and durable event storage.

AWS SQS in serverless flows

AWS SQS to decouple Lambda-based consumers and to level load in serverless architectures.

1

Determine throughput and delivery guarantees requirements

2

Select appropriate broker or managed service

3

Introduce schema registry and monitoring, implement consumers

⚠️ Technical debt & bottlenecks

  • Quickly implemented retry logic without idempotency checks
  • Growing dependence on proprietary broker features
  • Missing schema versioning for messages
Persistence I/ONetwork latencyConsumer throughput
  • Using a queue for immediate user wait times instead of direct communication
  • Persisting sensitive data in messages without encryption
  • Excessive partitioning without a consistency strategy
  • Undersized broker capacity leads to backlogs
  • Lack of observability hampers problem diagnosis
  • Unclear delivery semantics between components
Fundamentals of distributed systemsOperating and monitoring messaging infrastructureDesigning for fault tolerance and retries
Throughput requirementsDelivery guarantees and consistency requirementsOperational operability and monitoring
  • • Broker message count or size limits
  • • SLAs for delivery and latency
  • • Regulatory requirements for persistence and data protection