Gradient Clipping by Global Norm

easy
Unit 2 · Transformers & Training0/3 to milestone

A single bad batch can produce a huge gradient that wrecks training. Gradient clipping rescales all gradients together when their combined size is too large.

Implement clip_grad_norm(grads, max_norm):

  • grads is a list of NumPy arrays (one per parameter, any shapes).
  • The global norm is ∑i∥gi∥2\sqrt{\sum_i \|g_i\|^2} over every element of every array.
  • If it exceeds max_norm, multiply every gradient by max_norm / global_norm. Otherwise leave them as they are.
  • Return (clipped_grads, global_norm), where global_norm is the value before clipping. Don't modify the input arrays.
#training#optimization#numpy2 visible + 3 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Gradient Clipping by Global Norm · TechGradify