Catalog
concept#Data#Architecture#Integration#Observability

Streaming

An architectural paradigm for continuous processing and transmission of data streams in real time to enable low latency and loosely coupled integrations.

Streaming describes continuous processing and transmission of events or data streams in real time.
Established
High

Classification

  • High
  • Technical
  • Architectural
  • Intermediate

Technical context

Apache KafkaFlink / Spark Streaming / ksqlDBSchema registry (e.g., Confluent Schema Registry)

Principles & goals

Events are first-class; design around events.Decouple producers and consumers for scalability.Explicit state models and schemas for compatibility.
Build
Enterprise, Domain, Team

Use cases & scenarios

Compromises

  • Data inconsistencies with flawed retry strategies.
  • Cost increase due to persistent infrastructure and storage needs.
  • Operational overhead for schemas and backups.
  • Plan event evolution with backward-compatible schemas.
  • Use idempotent consumers and clear retry strategies.
  • Continuously measure metrics for latency, throughput and consumer lag.

I/O & resources

  • Event producers (microservices, devices, external systems)
  • Schema registry or data model
  • Streaming platform (broker, connectors, stream processors)
  • Continuous event streams for consumers
  • Real-time aggregations and metrics
  • Persistent event storage for auditing and replays

Description

Streaming describes continuous processing and transmission of events or data streams in real time. It enables low latency, continuous analytics and loosely coupled integration between producers and consumers and supports scaling, fault tolerance and stateful processing. Effective use requires architectural choices, operational practices and observability.

✔

  • Low latency for real-time decisions.
  • Improved scalability through asynchronous decoupling.
  • More flexible integration of heterogeneous systems.

✖

  • Increased operational effort for state management and replication.
  • More complex failure analysis and recovery scenarios.
  • Not always necessary for simple, latency-tolerant batch workloads.

  • End-to-end latency

    Time between production of an event and its processing by consumers.

  • Throughput (events/s)

    Number of events processed per second.

  • Consumer lag

    Delay or backlog between producer and consumer.

LinkedIn and Apache Kafka

LinkedIn developed Kafka for reliable distribution of event streams across services.

Netflix real-time streaming

Netflix uses streaming pipelines for monitoring, personalization, and operability in real time.

Industrial IoT telemetry

Factories stream sensor data for continuous analysis and predictive maintenance.

1

Define requirements: latency/throughput needs.

2

Establish event schemas and versioning rules.

3

Choose streaming platform and implement proof-of-concept.

4

Test and monitor stateful processing and fault tolerance.

5

Build runbooks, operational guides and observability.

⚠️ Technical debt & bottlenecks

  • Ad-hoc topic structures instead of clear naming conventions.
  • Insufficient archiving of old events and growth of storage.
  • Missing automated tests for event processing pipelines.
Network throughputPartition skewState size
  • Chasing real-time SLAs where batch solutions would suffice.
  • Direct writes from producers into consumer databases without events.
  • Uncontrolled topic creation without governance and cost assessment.
  • Underestimating the complexity of stateful operators.
  • Missing backpressure mechanisms under high load spikes.
  • Unconsidered schema incompatibilities between teams.
Understanding of distributed systems and partitioningKnowledge of state management and exactly-once processingOperational and observability skills for streaming platforms
Latency requirementsScalability and partitioningState management and storage strategy
  • • Limited storage for stateful operators
  • • Network bandwidth between compute zones
  • • Compatibility requirements for event schemas