Sample Ratio Mismatch Check

easy
Unit 7 · Economics0/3 to milestone

If an experiment was meant to split 50/50 but the arms got 10,000 vs 10,400 users, something may be broken (bots, redirects, logging). A sample ratio mismatch check runs a chi-square test on the counts.

Implement srm_check(counts, expected_ratios, alpha=0.001) returning (chi2, p_value, mismatch). For two arms (1 degree of freedom) the p-value is math.erfc(math.sqrt(chi2 / 2)); for more arms, compute the upper tail with the series for the regularised incomplete gamma function (as in the chi-square goodness-of-fit problem) with k - 1 degrees of freedom. mismatch is p_value < alpha.

#ab-testing#data-quality#hypothesis-testing2 visible + 1 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Sample Ratio Mismatch Check · TechGradify