Token

org.llm4s.context.tokens.Token

case class Token(tokenId: Int)

Represents a single BPE token as an integer vocabulary index.

The integer corresponds to the token identifier used by the underlying jtokkit / TikToken encoding (e.g. cl100k_base for GPT-4).

Attributes

Graph
Supertypes: trait Serializable

trait Product

trait Equals

class Object

trait Matchable

class Any
Show all

Returns a string representation of the object.

The default representation is platform dependent.

In this article

Generated with