Large language models do not read text exactly like humans. Before an LLM can process a sentence, the text is broken into smaller units called tokens.

This process is known as tokenization. It is one of the first steps that allows an LLM to convert human language into a format …