Reasoning models often reach the right answer long before they stop thinking. One cheap trick: periodically ask for an intermediate answer and stop once the same answer appears k times in a row.
Implement:
early_exit(answers, k, min_steps=0): answers is the list of intermediate answers at successive checkpoints (None when no answer could be parsed). Return the index of the checkpoint where the same non-None answer has just appeared k times consecutively, considering only indices ≥ min_steps. Return None if that never happens.token_savings(chunk_tokens, exit_index): chunk_tokens[i] is the number of reasoning tokens up to checkpoint i (since the previous one). Return the fraction of total tokens saved by stopping at exit_index, or 0.0 if it's None.exit_accuracy(runs, k): runs is a list of (answers, final_answer, correct_answer). Return (early_acc, full_acc), where the early answer is the answer at the exit point (falling back to final_answer if there's no exit).