Xavier and He Weight Initialisation

easy
Checkpoint challenge
Solve this to show you know the material and jump straight to Unit 2. Everything before it counts as cleared.

Initialise weights too large and activations explode. Too small and they vanish. The fix is to scale the random weights by the layer's size:

  • Xavier/Glorot (for tanh/sigmoid): normal with std 2fan_in+fan_out\sqrt{\dfrac{2}{\text{fan\_in} + \text{fan\_out}}}
  • He/Kaiming (for ReLU): normal with std 2fan_in\sqrt{\dfrac{2}{\text{fan\_in}}}

Implement init_weights(fan_in, fan_out, method, rng), returning a (fan_in, fan_out) array drawn with rng.normal(0, std, size=...), where method is "xavier" or "he". Raise ValueError for anything else.

Then implement activation_std_after(n_layers, width, method, activation, seed=0): push a batch of 256 standard-normal inputs through n_layers dense layers (no biases) initialised with init_weights (one np.random.default_rng(seed) for both inputs and weights), applying "relu" or "tanh" after each layer. Return the final activations' standard deviation.

#initialization#training#numpy2 visible + 3 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Xavier and He Weight Initialisation · TechGradify