org.llm4s.identity.TokenizerId
See theTokenizerId companion object
Identifies a BPE tokenizer vocabulary by its canonical name.
Tokenizer IDs are used by context-window estimation logic to select the correct byte-pair-encoding vocabulary for a given model, so that prompt and completion token counts are accurate without calling the provider API. The mapping from ModelId to TokenizerId is maintained by the context package.
Value parameters
- name
-
Tokenizer vocabulary name as used by tiktoken and related libraries (e.g.
"cl100k_base"for GPT-4 / GPT-3.5).
Attributes
- Companion
- object
- Graph
-
- Supertypes
-
trait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass Any
Members list
In this article