Multi Head Attention & Attention Specialization
In a multi-head attention mechanism, you observe that certain attention heads consistently focus on local token patterns (e.g., adjacent words), while others capture long-range semantic relationships. What is the architectural reason for this behavior, and how does it relate to the transformer's design?
Multi-head attention works because each head has independent linear projections (Q, K, V) for query, key, and value vectors. These projections are randomly initialized and learn different representations during training. This allows heads to specialize: some learn to attend to adjacent tokens (useful for syntax), others to distant tokens (useful for semantics). No explicit constraint or manual role assignment is needed—specialization emerges from the learning dynamics. The transformer does not use separate layers for local vs. global patterns; all heads operate in parallel. Heads are not manually assigned roles or given different cache sizes. While each head operates on a subset of the embedding dimension, the optimizer does not force specific local/global role assignments—this is an emergent property.