H2O (Zhang et al., 2023) keeps the KV cache small by observing that a few "heavy-hitter" tokens receive most of the attention. It keeps the tokens with the highest accumulated attention, plus a window of the most recent tokens.
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial