SGDR decays the learning rate along a cosine curve, then jumps back up. Cycle iii lasts Ti=T0⋅miT_i = T_0 \cdot m^iTi=T0⋅mi steps.
Within a cycle, at position tcurt_{cur}tcur: η=ηmin+12(ηmax−ηmin)(1+cos(π tcur/Ti))\eta = \eta_{min} + \tfrac12(\eta_{max} - \eta_{min})(1 + \cos(\pi\,t_{cur}/T_i))η=ηmin+21(ηmax−ηmin)(1+cos(πtcur/Ti)).
Implement sgdr_lr(step, lr_max, lr_min, T0, mult=1) for integer step >= 0.
sgdr_lr(step, lr_max, lr_min, T0, mult=1)
step >= 0