Checking an A/B test's p-value every day and stopping as soon as it dips below 0.05 inflates the false-positive rate far beyond 5%. Prove it by simulation.
Implement:
two_prop_p(conv_a, n_a, conv_b, n_b): the two-sided p-value of a pooled two-proportion z-test, . Return 1.0 when the pooled rate is 0 or 1.simulate_peeking(n_max, check_every, rate, n_sims, rng, alpha=0.05): run n_sims A/A tests (both groups have the same true conversion rate). For each simulation, draw rng.random((2, n_max)) < rate, then check after every check_every users per group (check_every, 2 * check_every, …, n_max). Count the simulation as a false positive if any check has p < alpha. Return the false-positive rate.pocock_alpha(k): a rough per-look threshold that keeps the overall rate near 5% for k equally spaced looks. Use the approximation 0.05 * math.log(1 + (math.e - 1) / k), which is the Lan–DeMets Pocock-type spending at the first look.