Residual Block Forward and Backward

A residual block computes $y = x + F(x)$, where $F(x) = W_2\,\text{relu}(W_1 x + b_1) + b_2$. The skip connection lets gradients flow straight through: $\frac{\partial y}{\partial x} = I + \frac{\partial F}{\partial x}$.

For a batch x of shape (N, D) with W1: (D, H), b1: (H,), W2: (H, D) and b2: (D,) (row-vector convention: x @ W1):

• residual_forward(x, W1, b1, W2, b2) returns (y, cache)
• residual_backward(dy, cache) returns (dx, dW1, db1, dW2, db2)

Then implement gradient_norm_through_depth(depth, D, seed=0), which shows why skip connections matter. Stack depth blocks, each with weights drawn from rng.normal(scale=0.01, size=...) and zero biases (W1 then W2 per block, with H = D), run a forward pass on rng.normal(size=(8, D)) drawn first, backprop dy = ones, and return the norm of dx. With residuals it stays close to sqrt(8 * D) even for deep stacks.