Transformer architecture
Almost every modern LLM is a decoder-only transformer: a stack of self-attention and feed-forward blocks that predicts the next token from all previous tokens.
Stable: this rarely changes. Reviewed 17 Oct 2026.
Key ideas
- Self-attention lets each token weigh every earlier token. Cost grows quadratically with sequence length, which is why long context is expensive.
- Multi-head attention runs several attention patterns in parallel; GQA/MQA share key/value heads to shrink the KV cache.
- Positional encoding (RoPE in most open models) tells the model token order; RoPE scaling is how context windows get extended.
- Mixture-of-Experts (MoE) routes each token to a few expert FFNs, so total parameters are large but active parameters per token are small.
- Training objective is next-token prediction; everything else (chat, tools, reasoning) is layered on through post-training.
Common mistakes
- Parameter count alone says little about quality or cost; for MoE models look at active parameters.