Catalog
concept#Product#Analytics#Delivery#Observability

Experimentation

A methodical approach for evidence-driven decision making using controlled tests in product and organizational contexts.

Experimentation is a systematic approach to generate evidence for product and organizational decisions by running controlled tests and learning from measured outcomes.
Established
Medium

Classification

  • Medium
  • Business
  • Organizational
  • Intermediate

Technical context

Feature flag systems (e.g., LaunchDarkly)Analytics and tracking platformsData warehouse and BI tools

Principles & goals

Formulate a clear hypothesis before any measurement.Define metrics in advance and align them to business goals.Ensure statistical integrity and data quality.
Discovery
Enterprise, Domain, Team

Use cases & scenarios

Compromises

  • Lack of correction for multiple testing leads to false positives.
  • Focusing on short-term metrics instead of long-term value.
  • Operational complexity due to inconsistent implementations.
  • Pre-register tests and hypotheses to avoid bias.
  • Use guardrail metrics and fallbacks for negative impacts.
  • Prefer iterative tests over isolated large experiments.

I/O & resources

  • Hypothesis with clear expectation
  • Defined target and safety metrics
  • Technical infrastructure (feature flags, tracking)
  • Statistical evaluation and decision basis
  • Documented learnings and follow-up hypotheses
  • Rollout or rollback plan

Description

Experimentation is a systematic approach to generate evidence for product and organizational decisions by running controlled tests and learning from measured outcomes. It defines the design, execution, analysis and governance of experiments, including hypothesis formulation, metrics selection and statistical interpretation. It reduces uncertainty and guides prioritization across discovery and optimization.

✔

  • Reduction of uncertainty through data-driven decisions.
  • Faster learning and improved prioritization of investments.
  • Measurable validation of assumptions and product changes.

✖

  • Requires sufficient traffic for statistical significance.
  • Not all hypotheses can be tested in a controlled manner.
  • Results can be confounded by external factors.

  • Conversion rate

    Percentage of desired actions within a cohort.

  • Lift (effect size)

    Relative change of the target metric between test and control groups.

  • Statistical power

    Probability of detecting a true effect.

E-commerce checkout experiment

A/B test of different payment layouts to increase conversion rate.

Landing page variant test

Measuring click and signup rates for alternative headlines.

Canary for release pipeline

Gradual rollout of an infrastructure change with metric checks.

1

Formulate hypothesis and define metrics.

2

Implement and instrument variants technically.

3

Allocate traffic, plan duration and set up monitoring.

4

Check data quality and perform statistical analysis.

5

Document results and execute decision.

⚠️ Technical debt & bottlenecks

  • Legacy tracking with inconsistent events.
  • Missing feature flag standards in codebase.
  • Insufficient data pipelines for nearline analyses.
Statistical powerTraffic availabilityImplementation effort
  • Aborting tests at first positive fluctuation without replication.
  • Choosing unsuitable metrics (e.g., vanity metrics).
  • Using experiment results outside the defined context.
  • Underestimating required sample sizes.
  • Ignoring seasonality and external effects.
  • Unclear assignment of users across devices.
Basic statistics and test designData instrumentation and trackingDomain knowledge for hypothesis formation
Data quality and accessibilityFeature flagging and orchestrationMeasurability and analysis capabilities
  • • Limited traffic for valid results.
  • • Legal and privacy constraints (e.g., GDPR).
  • • Technical limits for feature rollouts.