Training on hard one-hot labels pushes models to be over-confident. Label smoothing spreads a little probability to the other classes: with smoothing and classes, the target for the true class is , and every other class gets .
Implement smoothed_targets(labels, n_classes, eps) and smoothed_cross_entropy(logits, labels, eps=0.1): the mean over the batch of , using a numerically stable log-softmax. Also return the gradient with respect to the logits, which is (softmax - targets) / N: (loss, grad).