Llama, Mistral and most modern LLMs swapped two classic pieces for cheaper or better ones:
Implement silu(z), rms_norm(x, g, eps=1e-6) and swiglu_mlp(x, W1, W3, W2), then llama_block_mlp(x, g, W1, W3, W2), the residual sub-block x+SwiGLU(RMSNorm(x))x + \text{SwiGLU}(\text{RMSNorm}(x))x+SwiGLU(RMSNorm(x)).
silu(z)
rms_norm(x, g, eps=1e-6)
swiglu_mlp(x, W1, W3, W2)
llama_block_mlp(x, g, W1, W3, W2)