Aggregate an Eval Suite with Error Bars

easy
Unit 5 · Evaluation & Embeddings0/3 to milestone

An eval harness produces one record per example: {"task": name, "correct": 0 or 1}. Leaderboards report a single number, but without error bars you can't tell whether a 1-point gain is real.

Implement aggregate(records) returning:

  • "tasks": {task: {"n", "accuracy", "stderr"}}, with the binomial standard error sqrt(p(1 - p) / n)
  • "micro": accuracy over all examples pooled
  • "macro": the unweighted mean of task accuracies (each task counts equally, however many examples it has)
  • "macro_stderr": sqrt(sum(stderr_t^2)) / T for T tasks

Then implement significant_gain(a, b, z=1.96) for two aggregate outputs: True if b["macro"] - a["macro"] exceeds z * sqrt(a_se^2 + b_se^2) using the macro standard errors.

#evaluation#statistics#benchmarks2 visible + 1 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Aggregate an Eval Suite with Error Bars · TechGradify