org.llm4s.identity
package org.llm4s.identity
Members list
Type members
Classlikes
@Stable
Identifies a BPE tokenizer vocabulary by its canonical name.
Identifies a BPE tokenizer vocabulary by its canonical name.
Tokenizer IDs are used by context-window estimation logic to select the correct byte-pair-encoding vocabulary for a given model, so that prompt and completion token counts are accurate without calling the provider API. The mapping from model names to TokenizerId is maintained by the context package.
Value parameters
- name
-
Tokenizer vocabulary name as used by tiktoken and related libraries (e.g.
"cl100k_base"for GPT-4 / GPT-3.5).
Attributes
- Companion
- object
- Supertypes
-
trait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
object TokenizerId
Attributes
- Companion
- class
- Supertypes
-
trait Producttrait Mirrorclass Objecttrait Matchableclass Any
- Self type
-
TokenizerId.type
In this article