org.llm4s.identity.TokenizerId
See theTokenizerId companion object
@Stable
Identifies a BPE tokenizer vocabulary by its canonical name.
Tokenizer IDs are used by context-window estimation logic to select the correct byte-pair-encoding vocabulary for a given model, so that prompt and completion token counts are accurate without calling the provider API. The mapping from model names to TokenizerId is maintained by the context package.
Value parameters
- name
-
Tokenizer vocabulary name as used by tiktoken and related libraries (e.g.
"cl100k_base"for GPT-4 / GPT-3.5).
Attributes
- Companion
- object
- Graph
-
- Supertypes
-
trait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass Any
Members list
In this article