Chapter 7 - Building AI That Survives Reality

The Crux

Training a model is the beginning, not the end. Real AI systems must survive production: user drift, data drift, adversarial inputs, scaling, cost constraints. This chapter is about the unglamorous, essential work of making AI reliable.

Monitoring Model Drift

You deploy a model. It works. Six months later, it fails. What happened?

Data Drift

Definition: The input distribution changes.

Example: You trained a spam classifier on 2020 emails. In 2024, spammers use new tactics (crypto scams, AI-generated text). Your model hasn't seen these patterns.

Detection: Monitor input feature distributions. Alert if they shift significantly (KL divergence, Kolmogorov-Smirnov test).

Concept Drift

Definition: The relationship between inputs and outputs changes.

Example: A model predicts housing prices based on interest rates, location, etc. Then a recession hits. Same inputs now predict different prices.

Detection: Monitor model performance over time. If accuracy drops, you have concept drift.

Label Drift

Definition: The distribution of outputs changes.

Example: You trained a sentiment classifier on product reviews. Initially, 80% positive. Now, a bad product launch skews reviews to 60% negative. Model was calibrated for 80% positive.

Detection: Monitor predicted label distributions. Compare to historical baselines.