Catalog
concept#Machine Learning#Artificial Intelligence#Analytics

Inference

Inference is the application of a trained model to new data to produce predictions or decisions. It focuses on latency, scalability and resource optimization for production use.

Inference is the process of applying a trained machine learning model to new data to produce predictions or decisions.
Established
Medium

Classification

  • Medium
  • Technical
  • Architectural
  • Intermediate

Technical context

Feature store for consistent feature deliveryObservability stack (tracing, metrics, logging)CI/CD pipeline for model and infrastructure deployments

Principles & goals

Separation of training and inference pipelinesDefine measurable service levels for latency and throughputOptimize models for the target environment (quantization, pruning)
Run
Enterprise, Domain, Team

Use cases & scenarios

Compromises

  • Low model generalization leads to incorrect predictions
  • Operationalizing without monitoring increases outage risk
  • Missing performance tests cause SLA violations
  • Version models and use reproducible artifacts
  • Introduce automated performance and regression tests
  • Configure resource limits and quotas for stability

I/O & resources

  • Trained model in suitable format (e.g. SavedModel, ONNX)
  • Feature transformations and preprocessing logic
  • Infrastructure for hosting and scaling
  • Predictions or probability scores
  • Monitoring metrics for latency and errors
  • Logs and audit trails for requests

Description

Inference is the process of applying a trained machine learning model to new data to produce predictions or decisions. It covers aspects such as latency, scalability, resource usage and model optimization for production deployments. Common use cases include real-time predictions, batch inference and on-device models.

✔

  • Fast operational decisions through optimized runtimes
  • Scalable delivery of predictions to many users
  • Efficient resource usage via model compression

✖

  • Dependence on model quality and training data
  • Complexity when meeting latency and scaling requirements
  • Deployment on constrained devices requires trade-offs

  • Latency (P95)

    Time to response at the 95th percentile.

  • Throughput (requests per second)

    Number of successfully processed inference requests per second.

  • Error rate

    Share of failed or erroneous inference calls.

Realtime recommendation service

An online shop runs a low-latency endpoint that scores user actions in real time.

Batch scoring for risk analysis

Banks run nightly batch inference over transaction histories for risk scoring.

On-device object detection

Cameras run locally quantized models for object detection without cloud connectivity.

1

Validate model and choose suitable export format

2

Optimize models for target platform (quantization/pruning)

3

Set up serving infrastructure and configure endpoints

4

Execute automated tests and load tests

5

Implement monitoring, alerting and canary rollouts

⚠️ Technical debt & bottlenecks

  • Unversioned models and lack of reproducibility
  • Legacy inference runtimes with known performance issues
  • Lack of automation for rollbacks and tests
Model size and memory footprintNetwork bandwidth for cloud solutionsCPU/GPU utilization and scheduling
  • Using an unverified model in critical decision processes
  • Ignoring latency requirements in real-time applications
  • Scaling by naive replication without load distribution
  • Underestimating preprocessing costs in production
  • Over-optimization without regression tests
  • Missing security checks for inference endpoints
Knowledge in model optimization and quantizationExperience with serving technologies and containerizationOperational monitoring and performance tuning
Expected latency requirementsScaling and throughput needsAvailable hardware and cost constraints
  • • Compliance with privacy and regulatory requirements
  • • Hardware limits on edge devices
  • • Availability of stable feature pipelines