A treatment can win in every subgroup yet lose overall. This is Simpson's paradox, and it happens when the groups have very different sizes or base rates. Ignoring it leads to genuinely wrong product and medical decisions.
You get rows of (group, variant, successes, trials). Implement:
success_rates(rows), returning {"overall": {variant: rate}, "by_group": {group: {variant: rate}}}.is_simpsons(rows, a, b): True if variant a has a strictly higher rate than b in every group but a strictly lower rate overall, or the reverse.The classic example (kidney stone treatments):
| group | A | B |
|---|---|---|
| small stones | 81/87 | 234/270 |
| large stones | 192/263 | 55/80 |
A wins in both groups, but B wins overall.