ServiceTier

org.llm4s.llmconnect.model.ServiceTier
@Stable

The processing tier a provider serves a request at: what it costs, how fast it answers and how readily it sheds the request under load (RFC 0008, #2726). Set it with CompletionOptions.withServiceTier. It changes what a call costs and how it is scheduled, not what it returns.

Each provider maps it onto its own tiers, and reports on Completion.adjustments what it cannot send as asked:

  • '''OpenAI''': service_tier auto, default, flex or priority. Azure OpenAI takes auto, default and priority; Flex is reported Dropped there.
  • '''Anthropic''': service_tier auto (Priority Tier when the organization has a commitment, else standard) or standard_only (Default). Priority is sent as auto, reported Adjusted; Flex has no counterpart and is reported Dropped.
  • '''Gemini''': serviceTier standard, flex or priority. '''Vertex AI''': the X-Vertex-AI-LLM-Shared-Request-Type header, flex or priority (Provisioned Throughput first, then that tier). '''Bedrock''': Converse's serviceTier.type, default, flex or priority. Where a provider names no tier unless asked, Auto is not sent and reported NotNeeded; on Vertex AI, so is Default.
  • '''Every other provider''' has no tier to choose, and reports a set tier Dropped.

Which models and accounts each tier is open to is the provider's: a provider may serve a request at a different tier than the one asked for (Priority over its limit is served at standard on Gemini and Vertex AI), and Flex may be shed under load, so a Flex caller should expect and retry 429 and 503 replies.

Attributes

Source
ServiceTier.scala
Graph
Supertypes
trait Enum
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all

Members list

Type members

Enum entries

case Auto extends ServiceTier

The provider's own choice: the tier set for the project or account, or its default.

The provider's own choice: the tier set for the project or account, or its default.

Attributes

Source
ServiceTier.scala
case Default extends ServiceTier

The standard tier, at the standard price and speed.

The standard tier, at the standard price and speed.

Attributes

Source
ServiceTier.scala
case Flex extends ServiceTier

Cheaper and slower, served when there is capacity to spare; may queue or be shed under load.

Cheaper and slower, served when there is capacity to spare; may queue or be shed under load.

Attributes

Source
ServiceTier.scala
case Priority extends ServiceTier

Faster and more reliable under load, at a higher price, where the account has access to it.

Faster and more reliable under load, at a higher price, where the account has access to it.

Attributes

Source
ServiceTier.scala