Skip to content

LLM-as-judge

A model grades outputs against a rubric. Cheap and scalable, but it must be calibrated against human labels.

Evolving: the basics hold, details shift. Reviewed 17 Oct 2026.

Key ideas

Prefer binary pass/fail per criterion over 1 to 10 scales; they are more consistent.
Give the judge the rubric, the input, the output and a reference answer when available; ask for reasoning before the verdict.
Measure agreement with human labels (precision and recall on fails) before trusting it.
Pairwise comparison (A vs B) works well for comparing prompts or models; randomise order to avoid position bias.

In code

python

JUDGE = '''Criterion: the answer cites only facts present in <context>.
Return JSON: {"reasoning": str, "pass": bool}'''

verdict = judge.run_sync(f'{JUDGE}\n<context>{ctx}</context>\n<answer>{ans}</answer>')

Common mistakes

Using the same model as generator and judge without checking self-preference bias.