An eval harness produces one record per example: {"task": name, "correct": 0 or 1}. Leaderboards report a single number, but without error bars you can't tell whether a 1-point gain is real.
Implement aggregate(records) returning:
"tasks": {task: {"n", "accuracy", "stderr"}}, with the binomial standard error sqrt(p(1 - p) / n)"micro": accuracy over all examples pooled"macro": the unweighted mean of task accuracies (each task counts equally, however many examples it has)"macro_stderr": sqrt(sum(stderr_t^2)) / T for T tasksThen implement significant_gain(a, b, z=1.96) for two aggregate outputs: True if b["macro"] - a["macro"] exceeds z * sqrt(a_se^2 + b_se^2) using the macro standard errors.