float16 can only represent magnitudes from about 6e-8 (smallest subnormal) to 65504. Small gradients underflow to zero; large activations overflow to infinity.
Implement:
fp16_report(x): cast to np.float16 and return a dict with overflow (count of finite values that became inf), underflow (count of non-zero values that became 0) and max_rel_error (largest relative error among values that stayed finite and non-zero)best_loss_scale(grads, candidates): the largest scale in candidates such that grads * scale causes no overflow in float16, or None