Your LLM Judge Needs a Human Anchor: Calibrating LLM-as-a-Judge

An LLM judge is a measuring instrument, and an uncalibrated instrument reports confident numbers that can be badly wrong. To trust one, compare its verdicts against a golden set graded by trained people, and require chance-corrected agreement (Cohen's kappa) that approaches what two of those people achieve with each other. Common targets are a kappa of 0.6 as acceptable and 0.8 as strong.

Raw agreement hides the problem. In the example below, a judge agrees with human graders 92.7% of the time, yet its kappa is 0.68, and it would report a 16.5% failure rate on a product whose true failure rate is 9.8%. This guide covers how to build the golden set, run the calibration loop (with tested code), fix a judge that fails, and keep it honest after launch.

Why do LLM judges go wrong?

LLM-as-a-judge works. In the study that established the method, GPT-4 judges agreed with human preferences over 80% of the time, about as often as humans agree with each other. But that result was on a general benchmark. Your judge runs on your product, your rubric, and your traffic, and it fails in five predictable ways:

1. A vague rubric. "Is this response helpful?" invites the judge's own taste. If your human graders can't apply the rubric consistently, the judge can't either.
2. Self-preference. Judges tend to favor outputs that read like their own. Using the same model family to generate and to judge is the most common avoidable mistake.
3. Percent agreement that flatters. When 90% of outputs pass, a judge that says "pass" every time scores 90% agreement and catches nothing. Researchers testing nine judge models found that judges with high percent agreement could still assign very different scores, which is why kappa, not percent agreement, is the alignment metric to use.
4. Judges agreeing with each other. A panel of three LLM judges that agree with one another is not evidence that they agree with people. Recent work shows judge panels can be consistently wrong together.
5. Traffic drift. The judge was checked on last quarter's outputs. Your product, prompts, and users have changed since.

The fix for all five is the same: a set of items with answers decided by trained people, and a habit of measuring the judge against it.

How do you build a golden set?

A golden set is a frozen sample of real outputs, each graded by trained people under your rubric, with disagreements resolved. Six decisions shape it:

1. Size. The golden set's size sets how precisely you can measure the judge. In simulations at a 10% failure rate, the 95% confidence interval on kappa is roughly:

With 100 items, a measured kappa of 0.70 could really be anywhere from about 0.5 to 0.9, which can't tell an acceptable judge from a strong one. Start at 300 per rubric, and aim for 600 when the decision matters.

2. Sample from production. Pull real outputs, not synthetic ones, across your traffic mix.

3. Stratify toward failures. If failures are 5% of traffic, a random sample is mostly easy passes. Oversample known failure types and hard cases, and record the weights so you can report rates for real traffic.

4. Two or three graders per item. Independent, without seeing each other's answers or the judge's. This gives you the number that matters most: how often trained humans agree with each other, which is the ceiling for any judge.

5. Adjudicate. Disagreements go to a senior grader who decides and writes down why. Those notes usually become rubric fixes.

6. Freeze and version it. Never tune the judge's prompt while looking at golden items, or you're grading on the answer key. Keep a separate development set for prompt work.

How do you run the calibration loop?

Four steps, repeated until the judge clears the bar:

1. Measure human–human agreement on the golden set. This is the bar.
2. Run the judge on the golden set, with a fixed prompt and temperature.
3. Measure judge–human agreement against the adjudicated labels, with a confidence interval.
4. Read every disagreement, fix the rubric or the judge, and go back to step 2.

The code for steps 1 and 3 is short. This runs as written with scikit-learn and NumPy:

import numpy as np
from sklearn.metrics import cohen_kappa_score, confusion_matrix

def calibrate(human, judge, labels, n_boot=2000, seed=0):
    """Agreement between an LLM judge and adjudicated human labels."""
    human, judge = np.asarray(human), np.asarray(judge)
    raw = float((human == judge).mean())
    kappa = cohen_kappa_score(human, judge, labels=labels)
    rng = np.random.default_rng(seed)
    boots = []
    for _ in range(n_boot):
        idx = rng.integers(0, len(human), len(human))
        boots.append(cohen_kappa_score(human[idx], judge[idx], labels=labels))
    lo, hi = np.nanpercentile(boots, [2.5, 97.5])
    cm = confusion_matrix(human, judge, labels=labels)
    return {"raw_agreement": raw, "kappa": kappa, "kappa_95ci": (lo, hi), "confusion": cm}

def human_baseline(rater_a, rater_b, labels):
    """The bar the judge has to clear: two trained humans on the same items."""
    return cohen_kappa_score(rater_a, rater_b, labels=labels)

Reading the output

Here is an illustrative result on 600 golden items, graded pass or fail:

• Raw agreement: 92.7%. Looks great on a slide.
• Kappa: 0.68, 95% CI 0.59–0.77. Acceptable on the point estimate, but the interval dips below 0.6.
• What it does to your dashboard: the judge fails 99 items; people fail 59. You'd report a 16.5% failure rate when the true rate is 9.8%.

The confusion matrix tells you which way the judge errs, and that matters more than the score. This one is harsh: it rarely misses a real failure (2 of 59), but it flags 42 good outputs as bad. That's tolerable for a launch gate, where false alarms cost reviewer time. It's misleading for a quality metric you show customers or your board.

When does a judge pass? Set the bar before you look: judge–human kappa within a small margin of human–human kappa, the lower end of the confidence interval above your minimum (often 0.6), and no unacceptable error in the confusion matrix, such as passing safety failures.

What if the judge fails?

Read the disagreements before changing anything. Many turn out to be rubric gaps, not judge errors. Then work down this list, re-measuring after each change:

1. Fix the rubric. Turn every adjudication note into an explicit rule with an example. Ambiguity you remove for the judge also raises human–human agreement.
2. Split the score into binary checks. "Rate helpfulness 1–10" calibrates poorly. "Does the answer cite the policy section? Yes or no" calibrates well. Run one judge call per criterion.
3. Compare rather than score. For quality judgments, pairwise comparison (A or B?) usually agrees with people more than absolute scores. Run both orderings to cancel position bias.
4. Change the judge's model family. Especially if it matches the generator's.
5. Correct the number instead of the judge. If the judge's errors are stable, you can adjust its reported rate using the confusion matrix from calibration:

In the example above, the catch rate is 57/59 and the false alarm rate is 42/541. Plugging in the judge's 16.5% returns 9.8%, the true rate. More rigorous versions of this idea, such as prediction-powered inference, combine a large judge-labeled sample with a small human-labeled one to give a corrected estimate with a valid confidence interval.

If none of these gets the judge over the bar, the criterion may need people. That's a legitimate answer, not a failure.

When should you recalibrate?

Whenever something the judge depends on changes:

The monthly check is the one teams skip, and it's the one that catches a judge silently going stale as your users and outputs change.

Where do people stay permanently?

LLM judges scale measurement; they don't define it. Five jobs stay human:

• Writing the rubric, which is a statement of what your product is supposed to do.
• Grading calibration sets, the reference everything else is measured against.
• Launch gates for high-stakes releases, where a wrong "pass" is expensive.
• Adjudicating the cases where judge and people disagree.
• Red-teaming, where the point is finding failures nobody wrote a criterion for.

Practitioners agree. In a 2026 survey of AI practitioners, the human contribution they relied on most was designing evaluation methodology, named by 47%, and limits on human feedback were the top-cited bottleneck in safety and alignment work.

How Akmon helps

Most eval tools assume the human labels already exist. Akmon supplies them:

• Golden sets and rubrics, built with you and graded by reviewers trained on your documentation, with agreement stats and adjudication notes.
• The same people, every release. Our model audit regrades a production sample and your golden set against the same rubric after each release, so the numbers stay comparable and regressions are flagged.
• A baseline you can defend: how right your model is today and how it fails, as a scorecard you can show a customer or a board.

We can build and grade a 300–600 item golden set with you, and hand back the human–human baseline your judge has to beat. Book a call.

FAQ

How do I calibrate an LLM-as-a-judge? Grade a golden set of real outputs with trained people, run the judge on the same items, and compare with Cohen's kappa and a confusion matrix. Fix the rubric or judge and repeat until judge–human agreement approaches human–human agreement.

What is a good Cohen's kappa for an LLM judge? Above 0.6 is commonly treated as acceptable and above 0.8 as strong. The more meaningful bar is how close the judge gets to agreement between two trained people on the same items.

How big should my golden set be? At least 300 items per rubric; 600 when the decision matters. Below about 200, the confidence interval on kappa is too wide to tell an acceptable judge from a strong one.

Why not just use percent agreement? Because it rewards a judge for agreeing on the easy majority. When 90% of outputs pass, a judge that always says pass scores 90% and is useless. Kappa corrects for agreement by chance.

Can I use the same model to generate and to judge? Avoid it. Judges tend to prefer outputs like their own. Use a different model family, and calibrate either way.

How often should I recalibrate? Whenever the judge, rubric, or product model changes, and monthly on a fresh production sample to catch drift.

Sources

• Koji: LLM-as-a-judge vs. human evaluation (original MT-Bench agreement result; judge panels agreeing without matching humans)
• Thakur et al., Judging the Judges (arXiv 2406.12624) (nine judges; kappa over percent agreement)
• FutureAGI: LLM-as-a-judge in 2026 (kappa targets; pairwise with both orderings; same-family warning)
• OneUptime: calibrating an LLM judge with Cohen's kappa (calibration set; compare with human–human agreement)
• Kargi et al., Conformal Elo estimation (arXiv 2606.13221) (prediction-powered inference for debiasing judge labels)
• Prolific: what AI practitioners told us (47% rely on humans for eval methodology)