A production computer-vision model reported its estimates with striking consistency. Too striking. Today its owners know exactly how much of that confidence was real — because the distributions say so.
A production pipeline in physical-goods computer vision — cameras over a processing line, a regression model estimating a quantity that drives real commercial decisions. The model had been trained on expert annotations and validated against held-out labels. Its numbers looked clean. Its consistency was praised.
But the pipeline had a structural property nobody had priced in: it was a closed loop. The model learned from human judgments, was evaluated against human judgments, and shipped estimates no external instrument ever checked. Nothing outside the loop could confirm the numbers meant what everyone believed they meant.
Consistency was being mistaken for accuracy — and inside a closed loop, there is no way to tell them apart.
Item-level review couldn't answer the question. Any single estimate looked plausible. The doubt lived at a level no spot-check can reach: the shape of the model's behavior as a whole.
Instead of auditing predictions one by one, we audited them as a population. Every model output and every ground-truth label was treated as a sample from a distribution — and the two distributions were compared directly: spread, center, and shape, scored with the standard statistical distances (PSI, Kolmogorov–Smirnov, Wasserstein-1).
The question stopped being "is this estimate right?" — unanswerable inside the loop — and became "does the model's behavior have the same statistical signature as the reality it claims to describe?" That question, the distributions can answer.
The signature of a closed loop. A model trained on cloned judgment collapses toward its own mean — narrow, confident, and displaced. The width gap is the collapse; the displacement is the offset. Neither is visible one prediction at a time.
Two findings, both invisible to item-level review.
First, the collapse. The model's outputs varied by roughly 2% while the ground truth it was trained to match varied by roughly 9%. The model wasn't estimating the quantity — it was regressing toward a safe center and reporting it with manufactured confidence. Four-and-a-half times less variance than reality is not precision. It is the absence of measurement.
Second, the offset. Across object categories with no anatomical or visual relationship to one another, estimates ran systematically low by a consistent margin — approaching twenty percentage points. Unrelated categories failing in the same direction rules out annotator idiosyncrasy. The cause was structural: a pipeline-level property, not a labeling error. That single distinction redirected the remediation from "retrain the annotators" to "repair the loop."