Model Calibration: Expected Calibration Error

A model that says "90% confident" should be right about 90% of the time. Calibration measures whether predicted probabilities match reality. It matters for fraud thresholds, medical risk and LLM confidence scores.

Expected Calibration Error: split predictions into n_bins equal-width confidence bins over $[0, 1]$. For each non-empty bin $b$, compare the average confidence with the accuracy:

Implement reliability_bins(confidences, correct, n_bins=10), returning a list of dicts {"lo", "hi", "count", "confidence", "accuracy"} for the non-empty bins, in order. Bins are [lo, hi), except the last, which includes 1.0. Then implement ece(confidences, correct, n_bins=10).