LLM judges comparing answers A and B tend to favour whichever is shown first. The standard check: ask twice with the order swapped and see whether the verdict flips consistently.
Each record is (id, verdict_ab, verdict_ba), where verdict_ab is the judge's choice ("first", "second" or "tie") when A was shown first, and verdict_ba when B was shown first.
Implement:
resolve(verdict_ab, verdict_ba): map to the underlying winner "A", "B" or "tie". The pair is consistent if both runs point to the same answer. Inconsistent pairs, and pairs where either run says tie, resolve to "tie".bias_report(records): a dict with "consistency" (the fraction of consistent pairs), "first_position_rate" (the fraction of all individual non-tie verdicts that picked whichever answer was shown first) and "win_rate_A" (the fraction of resolved verdicts that are A, counting ties as half).