Inter-rater Reliability

Inter-rater reliability is the degree to which independent graders assign the same scores to the same items, a measure of how consistent and trustworthy an evaluation is.

High inter-rater reliability indicates the grading criteria are clear and applied consistently; low agreement signals an ambiguous rubric or unreliable judges.

It is what validates an LLM judge or a human grading panel before their scores are treated as authoritative in a benchmark.