TokenizerMapping

Maps LLM model names to appropriate tokenizers for accurate token counting.

Different LLM providers use different tokenization schemes, and even within a provider, newer models may use updated tokenizers. This object provides the mapping logic to select the correct tokenizer for a given model.

==Supported Providers==

| Provider   | Model Pattern           | Tokenizer      | Accuracy |
|------------|-------------------------|----------------|----------|
| OpenAI     | gpt-4o, o1-*            | o200k_base     | Exact    |
| OpenAI     | gpt-4, gpt-3.5          | cl100k_base    | Exact    |
| OpenAI     | gpt-3 (legacy)          | r50k_base      | Exact    |
| Azure      | (same as OpenAI)        | (inherited)    | Exact    |
| Anthropic  | claude-*                | cl100k_base    | ~75%     |
| Ollama     | *                       | cl100k_base    | ~80%     |
| Unknown    | *                       | cl100k_base    | Unknown  |

==Model Name Formats==

The mapper accepts various model name formats:

• Plain: gpt-4o, claude-3-sonnet
• Provider-prefixed: openai/gpt-4o, anthropic/claude-3-sonnet
• Azure: azure/my-gpt-4o-deployment
• Ollama: ollama/llama2

An Azure deployment is matched on the OpenAI model name its text contains, spelled as OpenAI spells it (gpt-4o, gpt-4, gpt-3.5, gpt-3, o1-): azure/my-gpt-4o-deployment selects o200k_base, while azure/my-gpt4o-deployment and any other name that does not contain such text selects cl100k_base. Every Azure deployment is reported as exact.

==Accuracy Considerations==

For non-OpenAI models, token counts are '''approximations'''. Claude uses a proprietary tokenizer that may differ 20-30% from cl100k_base. Always check TokenizerAccuracy to understand the expected accuracy.