Tell two engineers to rate model outputs from 1 to 5, giving only the instruction "rate the quality of each response." Their distributions will differ substantially. "Quality" means different things to different people, so without a precise rubric, scores capture rater intuition instead of the property you care about.
This is the central issue with vague rubrics: their scores are neither repeatable nor actionable. Without independent score reproduction, you cannot know whether a change reflects model behavior or measurement variance. Vague criteria also hide which output attribute needs improvement when scores fall.
An effective rubric's anatomy
Reliable, actionable rubrics contain three parts: a property definition, an anchored scoring scale, and calibration examples.
The property definition states exactly what you measure. Not "quality," but "factual accuracy: whether output claims match the provided source text, with no additions beyond it." Keep it narrow enough that two reasonable readers agree on what qualifies.
The scoring scale gives each value a meaning for that property. For factual accuracy, a 5-point scale might say: 5, every claim is directly supported by source text; 4, all are supported but one is imprecise; 3, one claim is absent but plausible; 2, one claim contradicts the source; 1, multiple claims contradict it. Each number has a concrete meaning.
Calibration examples are real outputs, or constructed ones closely resembling them, labeled with the right score and its rationale. They anchor raters: for an ambiguous case, compare it with an example to calibrate judgment. They also test your judge prompt: run them through the judge and confirm correct scores before using it on new data.
How vague rubrics break down
Vague rubrics often make scores track surface features instead of the property you care about. Length is the common example. Without explicit criteria, human raters and LLM judges often rate longer outputs higher because they seem more comprehensive. If length is not addressed, scores mix length with quality, leaving the cause of aggregate changes unclear.
Another failure mode is scope drift. A rubric meant to "measure factual accuracy" gradually gets applied to tone and formatting, as reviewers naturally use their overall judgment. Unless anchors exclude tone and formatting, the scope changes and historical scores are no longer comparable with current ones.
A third failure mode is poor inter-rater reliability on ambiguous cases. Every property has outputs whose correct score is genuinely unclear. A good rubric gives explicit direction, such as "if the claim is not in the source but is a logical inference from it, score as 3." A vague rubric leaves such cases to personal judgment, increasing variance at distribution tails.
LLM-as-judge rubric design
When a language model is the judge, the rubric becomes the judge prompt. The same principles apply, plus one constraint: the judge cannot ask clarifying questions. An ambiguous rubric is resolved according to the model's training predispositions, which may differ from your intent.
For LLM judges, the most reliable structure we have found is a one-sentence property definition, a concise scoring scale with numbered criteria, and two to three short calibration examples with scores inline. Examples do most of the work, anchoring interpretation and showing how to handle common edge cases.
Keep judge prompts focused on one property. A judge assessing five properties at once usually scores each less well than five judges assessing one property each. Holding five rubrics in mind degrades attention to each criterion. It costs more API calls, but better score quality and interpretability justify that cost for important properties.
Calibrate a rubric before trusting it
An uncalibrated rubric is not trustworthy. Calibration means labeling a sample with your rubric, having a second person label it independently, computing agreement, and examining disagreements to find ambiguity.
For human raters, Cohen's kappa above 0.7 is a reasonable reliability threshold. Below it, clarify the rubric before using it as a quality signal. Disagreements below the 0.7 threshold show what is underspecified: inspect those cases, find the output feature causing disagreement, and add explicit guidance.
For LLM judges, calibrate by running labeled samples through the judge and comparing its results with your labels. A judge agreeing 75% of the time but systematically missing a pattern needs guidance for that pattern. Add an example, re-run, and iterate until agreement exceeds your threshold.
Keep a version-controlled calibration set of at least 20 to 30 examples per rubric beside the rubric. After an update, re-run the set before deploying the new judge prompt. This catches improvements to ambiguous cases that accidentally harm cases previously scored correctly.
Rubric granularity: what level helps
Rubric granularity trades information for scoring cost. Ten dimensions reveal which output aspects improve or worsen, but need ten times more judge calls and ten times more numbers to interpret. A single dimension costs less and is easier to act on, but can miss important quality aspects.
A practical solution is a two-tier rubric. The primary rubric measures the key property for the prompt, the one whose decline would make you roll back a change. Run it on every eval and every CI check. Secondary rubrics cover supplementary properties, such as tone, formatting, and helpfulness, and run periodically rather than on every commit.
Your primary rubric should be trusted enough to serve as a deployment gate. It must be specific, calibrated, and stable. Secondary rubrics are exploratory: they show how the model changes across dimensions but do not block deployments.
How rubric quality affects decisions
Rubrics are practical infrastructure, not academic work. They drive choices about prompt changes, model upgrades, and production quality. Noisy scores create noisy decisions: you may roll back shippable changes or ship changes that need rollback because real score changes are indistinguishable.
Improving rubric quality improves decision reliability. It is unglamorous infrastructure, but it separates teams that can act on eval results from teams that run evals and debate what the numbers mean.