Multi-head Attention Block

Attention Block

Scaled dot-product attension

Each of $Q,K,V$ are linear projection of token embeddings.

$$\text{Attention(Q,K,V) = \text{softmax}}(\frac{QK^T}{\sqrt{d_k}})V$$

Multihead Attention

Now the same Attention block is applied mulitple time in parallel, then the result is concatenated.