Loading content
Concepts and practices for exposing trained machine learning models to production traffic, focusing on scalability, versioning and observability.
Model serving describes the systems and infrastructure that expose trained machine learning models to production traffic, handling scaling, versioning, routing and observability. It includes serving APIs, model lifecycle management and resource orchestration. The goal is reliable, low‑latency inference and reproducible deployment pipelines.
95th/99th percentile of response times to measure worst‑case latency.
Number of successful inference requests served per time unit.
Proportion of failed or erroneous requests relative to total traffic.
Use of TensorFlow Serving to deploy and version a CNN model with gRPC API.
KServe uses Knative/Ingress routing for canary rollouts and traffic splits.
Scaled offline predictions using ONNX Runtime as a Spark job for reporting.
Validate, serialize and register model in registry
Build serving image and deploy as a version
Configure routing, autoscaling and observability