A weight matrix's spectral norm (its largest singular value) bounds how much the layer can stretch its input, so it controls the network's Lipschitz constant. Spectral normalisation (Miyato et al., 2018) divides each weight by it, which stabilises GAN training.
Computing an SVD every step is too slow, so it's estimated with power iteration: repeat v = normalise(Wᵀu), u = normalise(W v), then σ ≈ uᵀWv.
Implement:
power_iteration(W, u0, iters=50): return (sigma, u, v)spectral_normalize(W, u0, iters=50): W / sigmalipschitz_bound(weights, u0s, iters=50): the product of the spectral norms of a list of layer weights (an upper bound on an MLP's Lipschitz constant with 1-Lipschitz activations like ReLU)