Gemini 2 Pro

Attention is a weighted lookup. Given a token, the layer computes how much attention to pay to every other token in the sequence, then produces a weighted sum of their values. Three projections per token: - **Query (Q):** what this token is looking for - **Key (K):** what this token offers others - **Value (V):** the information it contributes if attended to For each position, compute scores = Q · Kᵀ (dot product against every other token's key). Softmax normalizes these into weights. Output = weights · V. Multi-head attention runs this in parallel with different Q/K/V projections, so different "heads" learn to track different relationships — syntax, coreference, topic, etc. Concat and project back. The matrix view: if you have n tokens and d_model dims, Q/K/V are (n × d_k) each. Attention output is (n × d_k). With h heads, final output concatenates to (n × d_model).