Catalog
method#Software Engineering#Quality Assurance#DevOps

Automated Testing

Automated testing uses scripts and tools to run test cases repetitively and reliably, providing fast feedback on functionality and regressions.

Automated testing is a method to execute software tests with minimal human intervention, using scripts and tools to verify functionality, regressions and integration.
Established
Medium

Classification

  • Medium
  • Technical
  • Design
  • Intermediate

Technical context

CI systems (Jenkins, GitHub Actions, GitLab CI)Test data management toolsMonitoring and error reporting platforms

Principles & goals

Tests should be deterministic and independent.Test automation must be maintainable and well versioned.Fast feedback takes priority over complete coverage.
Build
Team, Domain

Use cases & scenarios

Compromises

  • Flaky tests lead to loss of trust in results.
  • Wrong prioritization can lead to high maintenance burden.
  • Over-reliance on tools can restrict flexibility.
  • Keep tests short and deterministic.
  • Isolate test data and provide it in a controlled way.
  • Plan regular maintenance and refactoring of the test suite.

I/O & resources

  • Source code and build artifacts
  • Test frameworks and libraries
  • Defined test cases and acceptance criteria
  • Test reports and metrics
  • Early defect identification
  • Regression test suite for continuous execution

Description

Automated testing is a method to execute software tests with minimal human intervention, using scripts and tools to verify functionality, regressions and integration. It accelerates feedback, increases repeatability and supports continuous delivery pipelines. Applied across unit, integration and end-to-end levels, it requires maintenance and test design discipline.

✔

  • Faster feedback on regressions and defects.
  • Increased consistency and reproducibility of tests.
  • Scalability of test runs in CI/CD pipelines.

✖

  • Initial setup requires effort and expertise.
  • Maintenance costs grow with test scope and system changes.
  • Not all tests are reliably automatable (e.g. UX).

  • Test coverage

    Measure of the portion of code exercised by automated tests.

  • Mean Time to Detect (MTTD)

    Average time until a regression test reports a defect.

  • Flakiness rate

    Share of intermittent tests that produce varying results without code changes.

Unit tests for backend service

A microservice uses unit tests to validate business logic on every commit.

Selenium-based UI testing

End-to-end tests automate user flows via Selenium in a staging environment.

API contract tests

Automated contract tests ensure API changes do not break consumers.

1

Prioritize test cases by risk and feedback need.

2

Select appropriate tools and integrate into CI.

3

Automate execution, monitor and continuously maintain tests.

⚠️ Technical debt & bottlenecks

  • Outdated test scripts without refactoring.
  • Monolithic test suites with long runtimes.
  • Unstructured test data and missing masking.
Flaky testsTest data provisioningLong test runtimes
  • Relying solely on automated UI tests as quality proof.
  • Unbounded growth of test suites without runtime optimization.
  • No maintenance of tests after major refactorings.
  • Misinterpreting flaky tests as code defects.
  • Introducing test automation too late in the project cycle.
  • Insufficient test data leads to false confidence.
Test automation scripting and framework knowledgeKnowledge of CI/CD and infrastructure automationTest design and defect analysis
Fast feedback for continuous deliveryTest isolation and deterministic executionRepeatable test environments (Infrastructure as Code)
  • • Limited test environments for realistic integration tests
  • • Budget for infrastructure and test tools
  • • Regulatory requirements for test data/masking