ALiBi: Attention with Linear Biases

ALiBi (Press et al., 2022) drops positional embeddings and instead adds a penalty to attention scores proportional to distance: query i attending to key j ≤ i gets a bias of $-m_h (i - j)$, with a different slope $m_h$ per head. Models trained this way extrapolate to longer contexts. It's used in BLOOM and MPT.