Attention on its own has no sense of word order. The original Transformer (Attention Is All You Need) fixes this by adding a positional encoding to each token embedding:
Implement positional_encoding(seq_len, d_model, base=10000.0) returning an array of shape (seq_len, d_model). Even columns use sine and odd columns use cosine, and each sine/cosine pair shares a frequency. d_model is always even. Vectorise it, with no Python loops.