Evaluate Embeddings on Semantic Similarity (STS)

easy
Checkpoint challenge
Solve this to show you know the material and jump straight to Unit 6. Everything before it counts as cleared.

STS benchmarks (part of MTEB) score sentence pairs from 0 to 5 by human judgement. An embedding model is good if the ranking of its cosine similarities matches the human ranking, measured by Spearman correlation.

Implement:

  • cosine_pairs(A, B): the row-wise cosine similarity between two (n, d) arrays
  • rankdata(x): ranks starting at 1, giving tied values the average of their ranks ([10, 20, 20, 30] → [1, 2.5, 2.5, 4])
  • spearman(x, y): the Pearson correlation of the ranks
  • sts_score(A, B, gold): Spearman between the cosine similarities and the gold scores
#embeddings#evaluation#statistics2 visible + 1 hidden tests
solution.py · Python 3.13
Loading editor…
Run your code to see results. ⌘↵ to run · ⌘⇧↵ to submit
Evaluate Embeddings on Semantic Similarity (STS) · TechGradify