Two models answered the same 500 test questions: A got 410 right, B got 420. Comparing accuracies as if they were independent samples throws away information. What matters is the questions where they disagree.
For paired 0/1 correctness arrays, let b01 = questions A got wrong and B got right, and b10 = questions A got right and B got wrong. Under the null hypothesis of equal accuracy, each disagreement is a fair coin flip.
Implement:
disagreements(a, b): return (b01, b10)mcnemar_exact(a, b): the two-sided exact binomial p-value: min(1, 2 * P(X <= min(b01, b10))) for X ~ Binomial(b01 + b10, 0.5). Return 1.0 if there are no disagreements.mcnemar_chi2(a, b): the continuity-corrected statistic (|b01 - b10| - 1)^2 / (b01 + b10) (0.0 when there are no disagreements)compare(a, b, alpha=0.05): a dict with "acc_a", "acc_b", "p_value" (exact) and "winner": "A", "B" or "tie" (tie when p ≥ alpha)