Evaluate Embeddings on Semantic Similarity (STS)

easy
Unit 5 · Evaluation & Embeddings0/3 to milestone

STS benchmarks (part of MTEB) score sentence pairs from 0 to 5 by human judgement. An embedding model is good if the ranking of its cosine similarities matches the human ranking, measured by Spearman correlation.

Implement:

  • cosine_pairs(A, B): the row-wise cosine similarity between two (n, d) arrays
  • rankdata(x): ranks starting at 1, giving tied values the average of their ranks ([10, 20, 20, 30] → [1, 2.5, 2.5, 4])
  • spearman(x, y): the Pearson correlation of the ranks
  • sts_score(A, B, gold): Spearman between the cosine similarities and the gold scores
#embeddings#evaluation#statistics2 visible + 1 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Evaluate Embeddings on Semantic Similarity (STS) · TechGradify