A model that can say "I don't know" trades coverage (how often it answers) for risk (how often its answers are wrong). Rank predictions by confidence and answer only the top ones.
Implement:
entropy_confidence(probs): for each row of class probabilities, 1 - H / log(K) where H is the entropy (in nats) and K the number of classes. Treat 0 log 0 as 0.risk_coverage(confidence, correct): sort by confidence descending (ties: original order). For k = 1…n, coverage is k/n and risk is the error rate among the top k. Return (coverages, risks) as arrays.aurc(confidence, correct): the mean of the risks (the area under the risk–coverage curve; lower is better)max_coverage_at_risk(confidence, correct, max_risk): the largest coverage whose risk is ≤ max_risk, or 0.0