Multi-Head Attention: Full Input Projection Not Slicing
Summary
A critical clarification about how multi-head attention works in Transformers: Each attention head receives the entire input embedding, not a slice of it.
The Common Misconception
Many explanations suggest that with 768 dimensions and 12 heads, each head gets a different 64-dimension...
I had multi-head attention wrong in my head for an embarrassingly long time. With 768 embedding dimensions and 12 heads, I assumed each head got its own 64-dimensional chunk of the input: head one takes dims 1-64, head two takes dims 65-128, and so on down the line. Tidy little picture. Wrong, though.
Every head sees the whole 768-dimensional embedding, all of it, every time. Each one has its own learned Q, K, and V weight matrices, and those project the full input down to 64 dimensions. All 12 heads run attention in parallel, each in its own 64-dim space. The outputs get concatenated (12 x 64 = 768 again) and a final linear layer mixes information across heads.
Where did my slicing idea come from? Probably the phrase "different parts of the feature space," which I somehow read as "different parts of the input," as if the embedding were a pizza and every head got a slice. It means learned representations. Twelve jurors hear the same testimony, and each one latches onto a different detail. You don't hand juror number seven a transcript with eleven twelfths redacted.
Doing it this way buys you cheap parallel 64-dim math plus 12 separately trained ways of looking at the same data. My sliced-up version would have been cheaper still, and dumber.