Rubric drift: measuring guideline change mid-project
The guideline edit nobody logged is the most common root cause we find, and the easiest to prove once the rubric is versioned.
How it happens
A project lead answers a reasonable question in a chat thread. The answer is correct, and it changes the operative definition of a label. Nothing in the pipeline records that anything happened.
Six weeks later the dataset contains two populations. Agreement within each is fine. Agreement across the boundary is not, and the project-level number splits the difference.
Finding the seam
Score every batch against every rubric version rather than only the one current when it was labelled. Where a batch agrees markedly better with a later version than the one it was labelled under, you have found the week the definition moved.
The fix is procedural
Make the rubric the only channel for definitional change, and make every change a version. The measurement above works because the versions exist; it cannot recover a seam in a project that only ever had one undated document.