> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/transformers/kv-cache.md).

# KV Cache

### Practical Implications of KV Cache

[https://epoch.ai/gradient-updates/how-has-deepseek-improved-the-transformer-architecture#:\~:text=What%20is%20the%20KV%20cache%20and%20why%20does%20it%20matter%3F](#practical-implications-of-kv-cache)

Store the key value for past tokens when auto regressively generating tokens. This is called KV Cache.

Size of KV cache per token = 2 (for key and value) \* attention head dim \* number of attention heads \* number of transformer blocks

if 1 byte for each float number, then for a KV cache size of 2.26M params, it's 4.7 MB per token.  

Now if you have let's say 100K of context size, then total cache size becomes = 470 GB of memory. 

That's around 140 ms of H100 time given the H100’s HBM bandwidth of 3.3 TB/s. The price per million tokens generated at $2 per hour per H100 would then be $80, around 5 times more expensive than Claude 3.5 Sonnet’s price to the customer (which is likely significantly above its cost to Anthropic itself).

**Hence, we would need to reduce this memory size by alot, using different techniques. --> Speculative Sampling with one of them**