llm4s-core
API
Generated with
llm4s-core/org.llm4s/org.llm4s.context/org.llm4s.context.tokens/StringTokenizer

StringTokenizer

org.llm4s.context.tokens.StringTokenizer
trait StringTokenizer

Converts a plain string into a sequence of Tokens using a specific BPE vocabulary.

Implementations are backed by jtokkit encodings and obtained via Tokenizer.lookupStringTokenizer; the interface is kept minimal to allow test doubles without a real encoding registry.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Abstract methods

def encode(text: String): List[Token]

Encodes text into a list of BPE Tokens.

Encodes text into a list of BPE Tokens.

Attributes

Generated with