Ford
Before you use the visualizer, you need to know one thing. Not everything. Just one thing. What a token is.
Marvin
A token is roughly four characters. Not a word. Not a letter. Something in between.
The word “run” is one token. Yes, it is three characters. I said roughly.
“run away” is two tokens. The space matters. Punctuation matters. Capitalization matters. These are not rounding errors. They are the unit of measurement.
Models do not read words. They predict from token streams. Each token shifts the probability of what follows. An AI response is a long chain of statistically likely next tokens. You experience it as intelligent conversation. It is an extraordinarily sophisticated next-word predictor.
They also charge by the token. You may not have noticed because you paid a flat monthly fee, or used a free tier, or work somewhere that bought a license. Someone is paying by the token. It is not always you. It is always someone. The flat fee products you are familiar with — ChatGPT, Claude, Gemini in a browser — are built on top of the same token-based pricing. The meter runs whether you see it or not.
Ford
See it yourself: OpenAI tokenizer tool. Type something in. Watch how it gets cut up. That is what the model actually sees.
Marvin
That is sufficient. You may now use the visualizer without embarrassing yourself.
Go to the Visualizer →