The L1 penalty λ‖w‖₁ drives many weights to exactly zero, which does feature selection. It isn't differentiable at 0, so plain gradient descent struggles. ISTA (proximal gradient descent) takes a gradient step on the smooth loss and then applies the L1 proximal operator, soft-thresholding:
Implement:
soft_threshold(z, t)fit_l1_logistic(X, y, lam, lr=0.1, iters=2000): minimise the mean log loss plus lam * ‖w‖₁ (the intercept isn't penalised). Each iteration: w = soft_threshold(w - lr * grad_w, lr * lam) and b -= lr * grad_b. Start from zeros and return (b, w).