Catalog
tool#Data#Analytics#Cloud#Platform

Databricks

Cloud-native platform for data engineering, analytics and collaborative work with Apache Spark.

Databricks is a unified, cloud-native data engineering and analytics platform that combines Apache Spark with managed infrastructure and collaborative workspaces.
Established
High

Classification

  • High
  • Technical
  • Technical
  • Advanced

Technical context

Cloud Storage (S3, ADLS, GCS)Delta Lake / data catalogsCI/CD tools (GitHub, Azure DevOps)

Principles & goals

Optimize for cloud-native usageModel data as a single source of truthEnsure automation and reproducibility
Build
Enterprise, Domain, Team

Use cases & scenarios

Compromises

  • Vendor lock-in when using proprietary features
  • Lack of cost control without governance
  • Misconfiguration can lead to data leaks
  • Make jobs idempotent and versionable
  • Use cost alerts and quotas for budget control
  • Use Delta Lake for transactions and time travel

I/O & resources

  • Raw data in cloud storage (S3, ADLS, GCS)
  • Access and security policies
  • Team structure and governance guidelines
  • Cleansed and catalogued datasets
  • Executable jobs and deployments
  • Monitoring data and operational metrics

Description

Databricks is a unified, cloud-native data engineering and analytics platform that combines Apache Spark with managed infrastructure and collaborative workspaces. It enables data engineering, data science, and analytics teams to build scalable ETL pipelines, run notebooks, and deploy machine learning workflows. Provided as a SaaS service across major clouds.

✔

  • Fast scaling of Spark workloads
  • Integrated collaboration for teams
  • Managed infrastructure reduces operational burden

✖

  • Costs can rise with poorly optimized workloads
  • Tight coupling with cloud providers required
  • Complex permission models in large organizations

  • Job runtime

    Average duration of ETL or notebook jobs.

  • Cost per dataset

    Compute cost relative to processed data volume.

  • Cluster utilization

    CPU/GPU and memory utilization over timeframes.

ETL pipelines for analytics

A company centralizes logs in S3, transforms them with Databricks and supplies cleansed data to BI systems.

Collaborative notebook analysis

Data science team uses shared notebooks for exploratory data analysis and collaborative visualization.

ML model serving with monitoring

Model training, versioning and production deployment are done via Databricks, including performance monitoring.

1

Set up cloud account and IAM roles

2

Configure network and storage access

3

Define workspace, cluster policies and permissions

4

Develop and test initial notebooks and ETL jobs

5

Establish cost monitoring, alerts and governance

⚠️ Technical debt & bottlenecks

  • Unrefactored monolithic ETL jobs
  • Outdated notebooks without tests or documentation
  • Hardcoded cloud credentials in scripts
Cost controlData latencyAccess management
  • Running production on small interactive clusters
  • Using platform as archive rather than an active data platform
  • Uncontrolled use of expensive GPU instances for simple jobs
  • Insufficient rights segregation leads to data access
  • Missing cost allocation complicates responsibilities
  • Overprovisioning clusters without autoscaling
Spark knowledge and distributed data processingCloud architecture and cost optimizationData modeling and ETL design
Scalability for large datasetsFast iteration for data scienceCompliance and data security
  • • Dependence on cloud providers
  • • Cost model and budget limits
  • • Regulatory data residency