What is a token?

When you are chatting with an LLM, the text often fades in like this:

That phrase is words, but tokens.

Different models would tokenize that text differently, actually:

The model typically generates one token at a time, selected from a pre-determined vocabulary. For example, GPT-2 has a vocabulary size of 50,257 tokens.