Swish (SiLU) and Mish

easy
Checkpoint challenge
Solve this to show you know the material and jump straight to Unit 2. Everything before it counts as cleared.

Smooth, non-monotonic activations often train better than ReLU in deep networks.

Implement:

  • swish(x) = x σ(x)x\,\sigma(x) (also called SiLU) and swish_grad(x) = σ(x)+x σ(x)(1−σ(x))\sigma(x) + x\,\sigma(x)(1-\sigma(x))
  • mish(x) = xtanh⁡(softplus(x))x \tanh(\text{softplus}(x)) where softplus(x)=log⁡(1+ex)\text{softplus}(x) = \log(1 + e^x), computed stably (use np.logaddexp(0, x))
#activations2 visible + 1 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Swish (SiLU) and Mish · TechGradify