R 0
A data-quality practice

Ground-truth standards, drift detection, and the human reference layer for computer vision and generative AI pipelines.

Request an audit Read the methodology

The reference point outside the loop.

Production CV pipelines PSI / KS / Wasserstein drift stack Ground-truth authorship Physical-world verification

Who calibrates the calibrators?

Instrument 01 — The verification divergence Functional automation Solving the obvious task Optimized throughput Rates faster, judges the same Judgment plateau More raters, no differentiation Reference authority Authors the standard the machines train on Value ■ Automation over time

As evaluation automates, item-level judgment commoditizes. The authority that defines the standard — the gold set, the rubric, the severity taxonomy — diverges. RaterZero operates on the diverging line.

How the practice helps

Four ways to hold the zero point.

01

Reference authorship

Gold sets, ground-truth dictionaries, rubric and severity-taxonomy design, edge-case adjudication. The standard your raters — and your auto-judges — are trained against.

Explore →
02 ★ Flagship

Drift & calibration audit

A fixed-scope statistical audit of your annotation exports: PSI, KS, and Wasserstein across geometry, class, and shape distributions. A report — not a rate card.

Request →
03

Eval architecture

Measurement design for pipelines: what to measure, holdout hygiene, gold-set expiry, and validating the LLM judge before it validates anything else.

Explore →
04

Accountability retainer

A named human reference for physical-ground-truth domains. Monthly calibration review and sign-off on ground-truth validity that survives an audit.

Explore →
Solved

The work is the evidence.

The variance collapse
A production model agreed with itself more than reality. We proved it in distributions — 2% output variance against the 9% it was built to match.
The catch
An indexing bug corrupted ground truth silently, past every automated layer. A calibrated human eye caught what the pipeline couldn't see.
Authoring the standard
Twelve object categories, one percentile-selected reference standard. An entire annotation team calibrated to a single fixed point.
The oracle run
An independent model, trained on expert annotations, auditing production from outside its own loop. The external referent, instantiated.
Similar challenge? Reach out →
By the numbers

Measured facts

~2%
Model output variance, measured
~9%
Ground-truth variance it should have matched
20pp
Systematic offset detected across unrelated categories
5
Statistical distance metrics in the drift stack
Field notes

Looking for clarity?

Article
Every pipeline has a patient-zero problem

Why annotation drift spreads through rater populations that lack a fixed reference — and why the cure is a point, not a process.
In preparation
Your gold set is a photograph, not a judge

How frozen references quietly expire while everything keeps calibrating against them.

In preparation
Who validates the LLM judge?

Why stacking automated evaluators creates a regress that only a human anchor can terminate.

In preparation
Behavioral cloning in a closed loop

What happens to a model that is trained, measured, and approved entirely inside its own distribution.

Instruments

Open tooling.

Drift Monitor

Statistical drift scoring for COCO segmentation exports — PSI, KS, JS divergence, and Wasserstein-1 across area, vertex count, aspect ratio, and compactness.

● Open source
Annotation Companion

A passive session mirror for annotators: dwell time, velocity, class distribution, and focus patterns. Self-reporting, never managerial.

● Open source
Oracle Run

An independent calibration model trained on expert annotation data — an external referent for auditing production models from outside their loop.

● In training