Rubric drift: measuring guideline change mid-project
ElenchLabsField noteMay 20, 2026
The guideline edit nobody logged is the most common root cause we find, and the easiest to prove once the rubric is versioned.
Research
Data quality is a measurement, not a promise
We think model performance is bounded by the data it learns from, and that the bound can be measured. These notes are what we measured, what changed because of it, and what each result does not establish.
The evidence behind six quality layers
Every layer on the site, with the published work it rests on and the limitation of that work, stated in the same place. Twenty-five references.
View the evidence
How the AI interviewer builds an expert's record
What the interview asks, why it never scores, and how a verified answer stops being asked. The same guide our experts read before they join a call.
Read the guideAll posts, newest first
Read post
Calibrating an LLM judge
ElenchLabsMethodApr 2, 2026
An LLM judge that has not been checked against human adjudication is a confident random number generator.
Why Cohen's kappa hides your worst annotator
ElenchLabsResearchMar 11, 2026
A project-level coefficient is an average, and averages are exactly the wrong tool for finding the one person dragging a dataset down.
6
Quality layers we research
Automated checks
Schema and format checks that run before any human sees a record. We study what they reliably catch, which is malformed rows and exact duplicates, and what they cannot: a well-formed label that is simply wrong.
