Catalog
concept#Data#Analytics#Platform

MapReduce

MapReduce is a programming model for distributed, parallel processing of large datasets. It separates computation into map and reduce phases and simplifies scaling, fault handling, and data partitioning across cluster environments.

MapReduce is a distributed programming model for parallel processing of large datasets across clusters; it abstracts map and reduce phases and enables horizontal scaling.
Established
High

Classification

  • High
  • Technical
  • Architectural
  • Intermediate

Technical context

HDFS or other distributed filesystemYARN / cluster resource managerObject storage (e.g., Amazon S3) for input/output

Principles & goals

Clear separation of map and reduce phasesPrioritize data locality to minimize network trafficDeterministic reducers and immutable input data
Build
Domain, Team

Use cases & scenarios

Compromises

  • Straggler tasks prolong overall job time
  • Network and shuffle bottlenecks with large data volumes
  • Lack of tuning can lead to high operational costs
  • Increase data locality via appropriate partitioning
  • Reduce shuffle volume with early aggregation
  • Handle skew via salting or custom partitioners

I/O & resources

  • Large structured or semi-structured datasets
  • Distributed storage system and network connectivity
  • Cluster resources (compute and storage nodes)
  • Aggregated or transformed datasets
  • Index or search data structures
  • Run statistics and audit logs

Description

MapReduce is a distributed programming model for parallel processing of large datasets across clusters; it abstracts map and reduce phases and enables horizontal scaling. It simplifies fault tolerance and data partitioning, making it suitable for batch analytics, index construction and large-scale aggregations. Implementations optimize locality, scheduling and resource utilization.

✔

  • Horizontal scaling of large data processing jobs
  • Built-in fault tolerance via task restarts
  • Simple programming model for complex aggregations

✖

  • High latency — primarily suitable for batch processing
  • Not optimal for iterative, low-latency workloads
  • Performance issues with data skew

  • Throughput (MB/s processed)

    Measures the amount of data processed per second and indicates scaling effects.

  • Job runtime (wall-clock)

    Total elapsed time of a MapReduce job from start to finish.

  • Straggler ratio

    Proportion of tasks that run significantly longer than the median and affect overall job time.

Google original paper

The original publication describes the MapReduce design and practical implementation details from Google's environment.

Apache Hadoop MapReduce

Official Hadoop implementation of the MapReduce paradigm, widely used in big-data ecosystems.

Batch analytics at large web companies

Typical case studies show MapReduce used for log analysis, index construction, and periodic aggregations in production environments.

1

Identify data sources and make them partitionable

2

Define map and reduce functions and test locally

3

Run jobs on a test cluster and adjust partitioning

4

Deploy to production with monitoring and resource tuning

⚠️ Technical debt & bottlenecks

  • Monolithic job pipelines without modularization
  • Hardcoded partitioning logic that hinders migration
  • Insufficient test coverage for failure cases and retry logic
Network/shuffle phaseDisk and I/O throughputData skew in partitions
  • Using it for low-latency streaming analytics instead of specialized stream engines
  • Running large jobs without tuning or monitoring causing resource contention
  • Using it for highly iterative algorithms instead of specialized frameworks
  • Underestimating shuffle costs on the network
  • Ignoring data skew when partitioning
  • Lack of observability for long-running tasks and stragglers
Knowledge of distributed systems and networkingExperience in map and reduce programming (e.g., Java, Python)Operational skills for cluster and resource tuning
Data volume and growth rateRequired parallelizability of jobsRequirements for fault tolerance and repeatability
  • • Input data must be partitionable
  • • Deterministic reduce operations required
  • • Dependency on a distributed storage system (e.g., HDFS, S3)