Evaluating an LLM-as-Judge (Cohen's κ)

medium
Unit 13 · Evaluation I0/2 to milestone

Teams increasingly use an LLM to grade model outputs. Before trusting it, you measure how well it agrees with human labels beyond chance using Cohen's kappa:

Pro problem

Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.

Start 7-day free trial
solution.py · Python 3.13
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Evaluating an LLM-as-Judge (Cohen's κ) · TechGradify