Bad initialisation or a too-high learning rate can leave ReLU units permanently off ("dead"), and tanh/sigmoid units stuck at their extremes. Monitoring activation statistics catches it early.
For a batch of pre-activations Z: (N, units), implement:
dead_units(Z): sorted indices of units whose pre-activation is ≤ 0 for every example (their ReLU output and gradient are always 0)relu_stats(Z): a dict with "dead_fraction" (the fraction of dead units) and "active_fraction" (the fraction of all unit-example entries that are > 0)tanh_saturation(Z, threshold=0.99): the fraction of entries where |tanh(z)| > threshold, where the gradient is nearly 0layer_report(layers): given a dict of layer name → pre-activation matrix, return the names whose dead fraction exceeds 0.5, sorted