OpenAICompatibleProvider

Registration for the generic openai-compatible provider: any endpoint speaking the OpenAI /chat/completions API, with no module and no code.

A named provider section with provider = "openai-compatible" needs a baseUrl and a model; apiKey and headers are optional (local servers need no key). It also accepts four provider-specific keys: contextWindow and reserveCompletion (whole numbers; the model's window cannot be known from an arbitrary endpoint, so these default to org.llm4s.llmconnect.config.OpenAICompatibleConfig.DEFAULT_CONTEXT_WINDOW and a quarter of the window up to org.llm4s.llmconnect.config.OpenAICompatibleConfig.DEFAULT_RESERVE_COMPLETION), registryProvider, and streamUsage (default true), which controls whether a streaming request asks for token usage with stream_options.include_usage; set it to false for a server that rejects that field.

'''Where the context window comes from.''' In this order: the section's contextWindow; else the model registry's entry for model under the registryProvider the section names (groq, together_ai, fireworks_ai, xai, perplexity, ...); else, when the section names none, the entry under the provider inferred from the baseUrl host; else the default. An explicit registryProvider is an instruction, so it switches the inference off. The host is inferred only by an exact match on the URL's host (api.groq.com, api.together.xyz, api.together.ai, api.fireworks.ai, api.x.ai, api.perplexity.ai), never by a substring, so a look-alike host such as api.groq.com.evil.example gets no registry window. Only the window is taken from the registry: reserveCompletion keeps its own rule (a quarter of the window, up to 2048) because a registry entry's output limit can be as large as the whole window. The lookup is strict - only the named provider's own entry for exactly that model id counts, never a substring match or another provider's entry for the same name - and a model the registry does not know, or lists with no input limit, gets the default. A registry window below the default is ignored too: many entries (most of Fireworks' older and newer models, Perplexity's retired ones) carry a 4096 placeholder for input, output and total alike, which would shrink an 80k-context model's prompt budget below the default and reject a reserveCompletion that fits it. The registry can therefore only enlarge the window, never shrink it; a model whose window really is smaller sets contextWindow. Several sections can use it side by side:

llm4s.providers {
 databricks {
   provider = "openai-compatible"
   baseUrl  = "https://ws.cloud.databricks.com/serving-endpoints"
   model    = "databricks-meta-llama-3-3-70b-instruct"
   auth {   // workload identity: no apiKey
     identityTokenFile = "/var/run/secrets/spiffe/svid.jwt"
     tokenUrl          = "https://ws.cloud.databricks.com/oidc/v1/token"
     clientId          = "<service principal UUID>"
     scope             = "all-apis"
   }
 }
 groq-main {
   provider = "openai-compatible"
   baseUrl  = "https://api.groq.com/openai/v1"
   model    = "openai/gpt-oss-120b"
   apiKey   = ${?GROQ_API_KEY}
   contextWindow = 131072
 }
 local-vllm {
   provider = "openai-compatible"
   baseUrl  = "http://localhost:8000/v1"
   model    = "Qwen/Qwen2.5-7B-Instruct"
 }
}

The client is OpenAICompatibleClient with the standard dialect: no reasoning parameters and no provider-specific decoding. A provider that needs those gets its own OpenAICompatibleDialect and descriptor in this module.