Embedding Layer Backward (Scatter-Add)

medium
Checkpoint challenge
Solve this to show you know the material and jump straight to Unit 12. Everything before it counts as cleared.

An embedding layer is a lookup: out = E[token_ids]. Its backward pass is easy to get subtly wrong: when a token appears several times in a batch, its gradient must be the sum of the gradients at every position. Plain fancy-index assignment dE[ids] += dout silently keeps only one of them.

Implement:

  • embedding_forward(E, ids): ids can have any shape, and the output has shape ids.shape + (dim,).
  • embedding_backward(dout, ids, vocab_size): return dE of shape (vocab_size, dim), accumulating correctly with np.add.at.
  • sgd_update(E, dE, lr): return the updated matrix (new array), touching only the rows that received gradient.
#embeddings#backprop#numpy2 visible + 2 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Embedding Layer Backward (Scatter-Add) · TechGradify