QA Evaluation: Exact Match and Token F1

easy
Unit 4 · Fine-Tuning & Evaluation0/2 to milestone

For question answering and RAG, answers rarely match the gold answer character for character. The SQuAD metrics normalise text first:

  1. lowercase,
  2. remove punctuation,
  3. remove the articles a, an and the as whole words,
  4. collapse whitespace.

Then:

  • Exact match: 1.0 if the normalised strings are equal, else 0.0.
  • F1: token-level overlap (counting repeated tokens with a multiset), F1=2PRP+RF1 = \frac{2PR}{P+R}. If either side has no tokens, F1 is 1.0 when both are empty and 0.0 otherwise.

Implement normalize_answer, exact_match and f1_score, then score(prediction, gold_answers), which returns the max EM and the max F1 over all gold answers: {"em": ..., "f1": ...}.

#evaluation#nlp#rag2 visible + 2 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
QA Evaluation: Exact Match and Token F1 · TechGradify