Feature Selection with Mutual Information

Mutual information measures how much knowing a feature reduces uncertainty about the label. Unlike correlation, it catches non-linear relationships.

For discrete arrays: $I(X; Y) = \sum_{x, y} p(x, y)\,\ln\frac{p(x, y)}{p(x)\,p(y)}$ (in nats), using empirical frequencies and skipping zero cells.

Implement:

• entropy(x): $-\sum p \ln p$
• mutual_information(x, y)
• select_k_best(X, y, k): the indices of the k columns with the highest MI with y, best first (ties: lower index)