Positional encoding and sequence order

Self attention is permutation invariant. Without positional information, the transformer cannot distinguish "the cat sat on the mat" from "mat the on sat cat the". Original Transformers used sinusoidal positional encodings; modern LLMs use rotary positional embeddings (RoPE) which rotate query/key vectors by a position