Leslie Smith's LR range test (popularised by fast.ai's lr_find) trains briefly while raising the learning rate exponentially, then picks a rate where the loss is falling fastest.
Implement:
lr_schedule(start, end, steps): steps learning rates growing geometrically from start to end, inclusivesmooth(losses, beta=0.98): an exponential moving average with bias correction: avg = beta * avg + (1 - beta) * loss and smoothed = avg / (1 - beta ** t) for t = 1, 2, …suggest_lr(lrs, losses, beta=0.98, diverge=4.0): smooth the losses, cut the curve at the first point where the smoothed loss exceeds diverge times the minimum smoothed loss so far, and return the learning rate where the smoothed loss falls most steeply with respect to log(lr) (use np.gradient)