Batch Normalisation: Train vs Inference

Batch normalisation normalises each feature over the batch, then applies a learned scale $\gamma$ and shift $\beta$:

The subtle part is that it behaves differently in training and inference.

Implement BatchNorm1d(num_features, momentum=0.1, eps=1e-5) with gamma (ones), beta (zeros), running_mean (zeros) and running_var (ones), and a method forward(x, training=True) for x of shape (batch, features):

• Training: normalise with the batch's own mean and (population, ddof=0) variance. Then update the running statistics: running = (1 - momentum) * running + momentum * batch_stat. Like PyTorch, use the unbiased variance (ddof=1) for running_var when the batch has more than one row.
• Inference: normalise with running_mean and running_var, and don't update them.