Tokens

What are Tokens?

Tokens are individual units of text used to represent words, characters, or subwords within the model's input. In natural language processing, text is broken down into tokens to facilitate analysis and processing by the model. The following image shows how an LLM separates a list into smaller tokens. Each time the color changes, that represents another token.

When does it matter?

Tokens matter because they affect the computational resources required to process text and can impact the model's performance. LLMs have a limit on the number of tokens they can handle due to computational constraints. If the input exceeds this token limit, it needs to be truncated, shortened, or split into multiple parts to fit within the model's capacity.

Context Window

The Context Window refers to the limited amount of text that an LLM (Large Language Model) considers when generating its responses. When an LLM generates text, it typically takes into account the surrounding context, which is a fixed-size window of tokens preceding the current position. Anything before the Context Window cut off, the LLM does not remember.

If the context window is set to 512 tokens, the model will have access to the 512 tokens preceding the current token being generated. The preceding tokens provide the necessary context for the model to generate coherent and contextually relevant responses.

Context windows play a crucial role in the performance and efficiency of LLMs. Setting an appropriate context window size is important to balance computational resources and the model's ability to understand and generate meaningful responses. If the context window is too small, the model may lack sufficient context to generate accurate or coherent responses. On the other hand, if the context window is too large, it can lead to computational challenges and may exceed the model's capacity.

Token Counter Tools

PreviousHallucinationsNextEthics

Last updated