Deleting outliers loses data. Winsorising caps them instead: values below the p-th percentile are raised to it, and values above the (100 − p)-th are lowered to it. This keeps one whale customer from dominating a mean.
Implement:
winsorize(x, lower=5, upper=95): clip at np.percentile(x, lower) and np.percentile(x, upper), ignoring NaNs when computing the percentiles and leaving NaNs in placewinsorized_mean(x, lower=5, upper=95): the mean of the winsorised values, ignoring NaNsfit_caps(train, lower, upper) and apply_caps(x, caps): compute the caps on training data only and apply them to new data, to avoid leakage