AI Quick Tips 317: Try OpenAI’s Tokenizer if you are still confused about tokens
If you’ve been using AI tools for a little while, you’ve probably heard “tokens” at some point. Tokens are how the different AI models break up sets of text. Similar to how humans break up sets of text into words.
