# Multi-head Attention Block

## Attention Block

<figure><img src="https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LFDuA0A2VRqmT31Blrq%2Fuploads%2FNVCW3tDfGBa2WWzgYBvf%2Fimage.png?alt=media&#x26;token=9c665643-6bff-4e01-96d0-d0d9ba4386c5" alt="" width="342"><figcaption><p>Scaled dot-product attension</p></figcaption></figure>

Each of $$Q,K,V$$ are linear projection of token embeddings.&#x20;

$$
\text{Attention(Q,K,V) = \text{softmax}}(\frac{QK^T}{\sqrt{d_k}})V
$$

## Multihead Attention

Now the same Attention block is applied mulitple time in parallel, then the result is concatenated.&#x20;

<figure><img src="https://1877261540-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2F-LFDuA0A2VRqmT31Blrq%2Fuploads%2FZAxfUA6zAJsb2tJq0pDO%2Fimage.png?alt=media&#x26;token=4e2e9cf2-8d67-4002-b605-b3e8a4952ec3" alt=""><figcaption></figcaption></figure>

