> For the complete documentation index, see [llms.txt](https://theshank.gitbook.io/ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://theshank.gitbook.io/ai/transformers/tokenziation.md).

# Tokenziation

Tokenization is a way to break down a string of continuous text into small individual chunks called tokens, each token can be a word/character/sub-word. 

Different tokenization schemes can be used based on different aspects. 

So let's say your tokenizer divided the whole text corpus into $$T$$ tokens i.e. there are your text is divided into total $$T$$chunks of smaller words, characters, sub-words, etc. **Now this** $$T$$ **becomes the Vocabulary size of your model.** 

Each token has an `token_id` $$\in \[1, T]$$. 

{% embed url="<https://huggingface.co/learn/nlp-course/chapter2/4>" %}