Using group theory to explore the space of positional encodings for attention

Attention is a computational primitive at the core of modern language models, allowing
internal representations to reference and influence each other. It’s how these models
handle sequential data in the first place.