Xavier and He Weight Initialisation

easy
Unit 1 · Backprop0/3 to milestone

Initialise weights too large and activations explode. Too small and they vanish. The fix is to scale the random weights by the layer's size:

  • Xavier/Glorot (for tanh/sigmoid): normal with std 2fan_in+fan_out\sqrt{\dfrac{2}{\text{fan\_in} + \text{fan\_out}}}
  • He/Kaiming (for ReLU): normal with std 2fan_in\sqrt{\dfrac{2}{\text{fan\_in}}}

Implement init_weights(fan_in, fan_out, method, rng), returning a (fan_in, fan_out) array drawn with rng.normal(0, std, size=...), where method is "xavier" or "he". Raise ValueError for anything else.

Then implement activation_std_after(n_layers, width, method, activation, seed=0): push a batch of 256 standard-normal inputs through n_layers dense layers (no biases) initialised with init_weights (one np.random.default_rng(seed) for both inputs and weights), applying "relu" or "tanh" after each layer. Return the final activations' standard deviation.

#initialization#training#numpy2 visible + 3 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Xavier and He Weight Initialisation · TechGradify