A sliding-window KV cache that drops the oldest tokens makes LLM quality collapse, because models dump a lot of attention onto the first few tokens ("attention sinks"). StreamingLLM's fix: always keep the first n_sink tokens, plus a rolling window of the most recent ones.
Unlock this and every premium problem across all tracks — plus hidden-test grading, advanced AI engineering modules and new content every week.
Start 7-day free trial