A model with ninety-two percent accuracy on a held-out test set can still be operationally useless. If the two classes in the problem are distributed ninety to ten, a model that predicts the majority class for every observation achieves ninety percent accuracy without learning anything about the problem. This is a well-known pathology, and most data scientists know to check for it. What they less often check for is the distribution of errors across the business-relevant segments that the model will actually be applied to in production.
A churn model that is accurate on average but substantially less accurate for the enterprise customer segment — the segment that represents eighty percent of revenue — is not a model that should be deployed to drive enterprise retention decisions. The aggregate validation metric conceals the slice-level failure. Proper model validation requires evaluating performance separately for every segment that the model's outputs will be used to make decisions about, and it requires defining acceptable performance thresholds for each segment independently, not just in aggregate.
Calibration is the validation dimension that determines whether a model's confidence scores can be used as inputs to decision logic. A model that outputs a churn probability of 0.8 for a customer should be correct about eighty percent of the time when it outputs that score. If the model is systematically over-confident — if customers it scores at 0.8 actually churn at a rate of 0.6 — then any decision logic that uses the raw score as a threshold is making decisions based on wrong information. Calibration plots and reliability diagrams are the standard diagnostic tools. Platt scaling and isotonic regression are the standard correction methods. Most deployed models skip this step. Most deployed models have calibration problems that are only discovered when the decision outcomes are worse than the validation metrics predicted.
Temporal validation is the test that distinguishes models that have learned generalizable patterns from models that have overfit to the specific characteristics of a historical training period. Walk-forward validation — training on data through month N, validating on month N+1, sliding the window forward, and evaluating performance across the full sequence — reveals how model performance degrades as the gap between training period and prediction period grows. A model that performs well with a two-week lag and poorly with a six-month lag is telling you something specific about the stability of the patterns it has learned. That information should inform the retraining schedule in production — not be discovered after six months of degrading model quality.