ServiceTier
org.llm4s.llmconnect.model.ServiceTier
@Stable
enum ServiceTier
The processing tier a provider serves a request at: what it costs, how fast it answers and how readily it sheds the request under load (RFC 0008, #2726). Set it with CompletionOptions.withServiceTier. It changes what a call costs and how it is scheduled, not what it returns.
Each provider maps it onto its own tiers, and reports on Completion.adjustments what it cannot send as asked:
- '''OpenAI''':
service_tierauto,default,flexorpriority. Azure OpenAI takesauto,defaultandpriority;Flexis reportedDroppedthere. - '''Anthropic''':
service_tierauto(Priority Tier when the organization has a commitment, else standard) orstandard_only(Default).Priorityis sent asauto, reportedAdjusted;Flexhas no counterpart and is reportedDropped. - '''Gemini''':
serviceTierstandard,flexorpriority. '''Vertex AI''': theX-Vertex-AI-LLM-Shared-Request-Typeheader,flexorpriority(Provisioned Throughput first, then that tier). '''Bedrock''': Converse'sserviceTier.type,default,flexorpriority. Where a provider names no tier unless asked,Autois not sent and reportedNotNeeded; on Vertex AI, so isDefault. - '''Every other provider''' has no tier to choose, and reports a set tier
Dropped.
Which models and accounts each tier is open to is the provider's: a provider may serve a request at a different tier than the one asked for (Priority over its limit is served at standard on Gemini and Vertex AI), and Flex may be shed under load, so a Flex caller should expect and retry 429 and 503 replies.
Attributes
- Source
- ServiceTier.scala
- Graph
-
- Supertypes
Members list
In this article
