Models Core
Token
The basic unit of text that language models process. Roughly 3/4 of an English word, or ~4 characters. 1K tokens ≈ 750 words.
The basic unit of text that language models process. Roughly 3/4 of an English word, or ~4 characters. 1K tokens ≈ 750 words.