The needle-in-a-haystack test hides one fact (the needle) at different depths inside contexts of different lengths, then asks the model to retrieve it. The results form a grid that shows where long-context recall breaks.
Implement:
build_haystack(filler, needle, n_sentences, depth): cycle through the filler sentences to make n_sentences sentences, then insert the needle at position round(depth * n_sentences) (depth 0 = start, 1 = end). Join with spaces.needle_position(haystack, needle): the needle's relative character position, index / len(haystack)summarise(results): results maps (length, depth) to True/False. Return a dict with "by_depth" (accuracy per depth), "by_length" (accuracy per length) and "worst": the (length, depth) of the first failure in sorted order, or None.