MLOps Foundations: Pipelines, Registries, Monitoring

MLOps is the operational discipline of training, deploying, monitoring, and retraining machine learning models reliably.

MLOps: Getting ML Models from Notebook to Production Without Wishful Thinking

MLOps applies the lessons of DevOps to machine learning systems, with the added complexity that the artifact you're deploying (a model) is a function of data that itself changes. A model that was excellent last quarter can silently degrade this quarter because the input distribution shifted, a feature pipeline broke, or a labeling process changed upstream. MLOps establishes the observability, reproducibility, and automation needed to catch and correct these issues before they cost you customers or money. It is less about specific tools (SageMaker, Vertex AI, MLflow, Weights & Biases) and more about disciplined workflows any of those tools can support.

The four workflows every ML system needs

(1) Data pipeline — ingesting raw data, transforming it into features, versioning both. (2) Training pipeline — reproducibly training a model from versioned data and code, producing a versioned model artifact with metrics. (3) Deployment pipeline — promoting a trained model to production with the same rigor as a service deploy (staging, canary, rollback). (4) Monitoring — tracking input distributions, prediction distributions, and downstream business metrics after deployment. Most teams build 1 and 2 well and neglect 3 and 4, which is why so many models decay unnoticed. Aim for parity across all four before scaling to more models.

Model and experiment tracking

Every trained model should be reproducible from: the code (git SHA), the data (dataset version / snapshot ID), the hyperparameters (config file), and the environment (container image or lockfile). MLflow, Weights & Biases, and Neptune all handle this well; the important thing is that the choice is standardized across the team. Metadata to log per run: metrics (train/val/test), hyperparameters, dataset ID, code SHA, hardware, wall clock. The rule: if you can't reproduce a model's training in one command from these artifacts, you don't really own it — you can't diagnose regressions or roll back with confidence.

Deployment patterns

(1) Batch inference — model runs on a schedule, writes predictions to a table. Simplest; appropriate when predictions don't need to be real-time (churn scores, next-day recommendations). (2) Online inference behind an API — model wrapped in a service, called synchronously. Requires latency and reliability engineering; monitor P95 latency and error rate as you would any service. (3) Embedded — model shipped inside the client (mobile, browser). Painful to update; use only when latency or privacy demands it. For online inference, adopt canary rollout and shadow mode: run the new model in parallel, log its predictions without serving them, compare distributions, then promote.

Monitoring beyond accuracy

Accuracy monitoring is delayed — you often don't know if a prediction was right for days or weeks. Real-time signals: (a) input drift — has the distribution of features shifted materially from training? Population Stability Index (PSI) or KL divergence per feature, alerted when they cross thresholds. (b) prediction drift — has the distribution of model outputs changed? Often faster to detect than accuracy loss. (c) downstream metric — the business KPI the model influences (conversion, click, retention). (d) infrastructure — latency, throughput, error rate. When accuracy ground truth arrives, retro-actively join it to predictions and update dashboards. Alert on drift, not on accuracy loss — by the time accuracy drops, damage is done.

Team structure and ownership

Small orgs (1-3 ML people): one team owns end-to-end from data to deployment. MLOps is 20-30% of everyone's job. Medium (4-15 ML people): split into an ML platform team (owns feature store, training infra, deployment tooling) and applied ML teams (own models for specific product areas). Large (15+): full MLOps platform team with PMs, plus specialized ML infra roles (feature engineering, serving infra, monitoring). Regardless of scale, avoid the anti-pattern of data scientists throwing notebooks over the wall to a separate 'ML engineering' team — the handoff kills iteration speed. Data scientists should ship models to production themselves, on a platform that makes it safe to do so.

Frequently asked questions

Do we need a feature store?
Below ~5 production models, no — the coordination overhead of a feature store exceeds its benefit. Above that, or when the same features are used for both training and online inference, a feature store pays off by ensuring training/serving parity and enabling feature reuse.
How often should we retrain?
It depends on drift, not on a schedule. Set up drift monitoring and retrain when input distributions cross a threshold or when downstream metrics degrade. Calendar-based retraining (weekly, monthly) is a rough proxy but wastes compute or misses drift depending on the model.
MLflow, Weights & Biases, or a cloud-managed platform?
For small teams, W&B or MLflow OSS on your own infra is enough. For larger teams or teams already on AWS/GCP/Azure, the cloud-managed platforms (SageMaker, Vertex AI, Azure ML) reduce operational load at the cost of some lock-in. The choice matters less than picking one and standardizing.

Related fundraising guides (40)

Investor directory · Fundraising library · Articles A–Z · Company funding database