Model drift vs. data drift
What is model drift?
Model drift, also known as concept drift, occurs when the relationship between input features and the target variable changes. In other words, the underlying patterns a model has learned no longer hold true in the real world. This can cause the model’s predictions to become less accurate, even if the input data seems valid.
Example:
A credit risk model trained during a stable economy may underperform during a recession, even if the input features (income, credit score) haven’t changed in distribution.
What is data drift?
Data drift refers to changes in the statistical properties of input data over time. Unlike model drift, which impacts the model’s understanding of the relationship between inputs and outputs, data drift involves shifts in the feature distributions themselves.
Example:
If the average age of users in your system gradually decreases, and your model was trained on an older population, this shift could cause degradation in performance.
Why it matters in AI/ML
Monitoring both types of drift is crucial for production ML systems:
- Data drift can signal upstream data pipeline issues or changing user behavior.
- Model drift often indicates that a retraining process or model revision is needed.
Neglecting either can lead to poor predictions, compliance risks, or business-impacting errors.
How to detect and address drift
- Detecting data Drift:
- Use statistical tests (e.g., KL divergence, PSI) on feature distributions.
- Monitor for schema changes, null value increases, or missing features.
- Detecting model drift:
- Track performance metrics over time (accuracy, precision, etc.)
- Compare real-world predictions against updated ground truth labels.
- Responding to drift:
- Recalibrate models or retrain with newer data
- Conduct error analysis to isolate affected segments
- Automate retraining triggers based on drift thresholds
Related
Want to go deeper? See related topics like AI data validation to learn how production teams address these challenges.