What a token is, and why Hindi takes more of them
What is a token, and why does the same sentence take more of them in Hindi?
An LLM reads your text as tokens: pieces cut from a fixed list. You pay per token, and the text a model can read in one request is counted in tokens. The same text usually takes more tokens in Hindi than in English. How many more depends on the tokenizer, the part that does the cutting.
