Mixture-of-Experts routers tend to collapse onto a few favourite experts. The Switch Transformer auxiliary loss pushes routing towards uniform:
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial