Semantic Deduplication of a Dataset

Fine-tuning datasets are full of paraphrased near-duplicates that exact matching misses. SemDeDup groups examples whose embeddings are nearly identical and keeps one per group.

Implement:

• duplicate_groups(E, threshold): link every pair with cosine similarity ≥ threshold, and return the connected groups (use union-find, so AB and BC puts A, B and C together). Each group is a sorted list of indices; sort the groups by their first index. Singletons are groups too.
• dedupe(E, threshold): the sorted list of kept indices, one per group (the lowest index)
• dedupe_rate(E, threshold): the fraction of examples removed