Mixture-of-Experts Top-k Routing

medium
Unit 11 · Transformers0/2 to milestone

Mixtral, DeepSeek and (reportedly) GPT-4 are Mixture-of-Experts models: each token is sent to only kk of EE expert networks, chosen by a small learned router.

Pro problem

Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.

Start 7-day free trial
solution.py · Python 3.13
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Mixture-of-Experts Top-k Routing · TechGradify