The KV cache: memory per token

During autoregressive generation, each token attends to all previously generated tokens. Recomputing key and value projections for every prior token at every step is O(n²) in sequence length. The KV cache stores these projections in GPU memory, reducing per step computation to O(n).