If benchmark questions leaked into training data, scores are inflated. The common check: look for long n-gram overlaps between each test item and the training corpus (GPT-3 used 13-grams).
Implement:
ngrams(text, n): the set of word n-grams after lowercasing and keeping only alphanumeric words (re.findall(r"[a-z0-9]+", text.lower())). Texts shorter than n words yield no n-grams.build_index(train_docs, n): the union of all training n-grams.contamination_report(test_items, index, n, threshold=0.5): for each test item, the overlap ratio = (its n-grams found in the index) / (its total n-grams), or 0.0 if it has none. Return {"contaminated": [indices with ratio >= threshold], "ratios": [...]}.