Worth noting that a token is not necessarily a word, though can be. One word could also take multiple tokens. It can also vary from LLM to LLM and their tokenization methods.
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments