In data-parallel training, every GPU computes gradients on its own batch, and then all GPUs need the sum. Ring all-reduce does this with each worker talking only to its neighbour, so bandwidth per worker stays constant as you add workers.
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial