SGD with Momentum and Nesterov

easy
Unit 3 · Optimization0/3 to milestone

Plain gradient descent zig-zags across narrow valleys. Momentum keeps a running velocity so consistent directions speed up and oscillations cancel out.

With learning rate η\eta and momentum μ\mu (PyTorch's formulation):

  • Momentum: v←μv+gv \leftarrow \mu v + g, then w←w−ηvw \leftarrow w - \eta v
  • Nesterov: v←μv+gv \leftarrow \mu v + g, then w←w−η(g+μv)w \leftarrow w - \eta (g + \mu v)

Implement sgd_step(w, g, v, lr, momentum=0.0, nesterov=False) returning (new_w, new_v) without modifying the inputs. Then implement minimize(grad_fn, w0, lr, momentum, nesterov, steps), which runs the optimiser from v = 0 and returns the list of positions, including w0.

#optimization#training2 visible + 2 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
SGD with Momentum and Nesterov · TechGradify