llm4s-docs
API
Generated with
llm4s-docs/org.llm4s/org.llm4s.llmconnect/org.llm4s.llmconnect.spi

org.llm4s.llmconnect.spi

package org.llm4s.llmconnect.spi

Members list

Type members

Classlikes

@Stable
final case class EmbeddingConfigSpec

What an embedding provider needs from its section, and what to use when the section omits it.

What an embedding provider needs from its section, and what to use when the section omits it.

The embedding counterpart of ProviderConfigSpec, with one structural difference worth knowing. Chat config is keyed by the user's ''instance'' name (llm4s.providers.my-openai.baseUrl), so a reference.conf fragment cannot express a per-provider default - it does not know the instance name. Embedding config is keyed by the ''provider id'' (llm4s.embeddings.openai), so a module's own reference.conf can and does bind that provider's environment variables.

The division is therefore: defaults live here, environment bindings live in the module's reference.conf. Before this existed the two were duplicated - reference.conf said baseUrl = "http://localhost:11434" and EmbeddingsConfigLoader said DefaultOllamaEmbeddingBaseUrl - and nothing kept them in step.

== The API key == The key is the section's own apiKey when it sets one, and otherwise the vendor's shared key, llm4s.credentials.<id>.apiKey, which the module's reference.conf binds to the vendor's variable (llm4s.credentials.openai.apiKey = ${?OPENAI_API_KEY}). The same shared key serves the vendor's chat sections, so one variable configures both. org.llm4s.config resolves that fallback before calling the descriptor; nothing here reads configuration.

Value parameters

apiKeyEnv

the variables the module's reference.conf binds to llm4s.credentials.<id>.apiKey, highest precedence first, named in the error when the key is missing. A declaration for the message only: the binding itself is the reference.conf line, and each module's round-trip spec proves the two agree. Empty when nothing binds the shared key, and the error then names the shared path instead.

defaultApiKey

stand-in for a provider that takes a key but does not need a real one - Ollama running locally.

defaultBaseUrl

base URL used when the section omits one.

defaultModel

model used when neither EMBEDDING_MODEL nor the section names one.

modelEnv

the variable the module binds to the model, named likewise.

requiresApiKey

the provider cannot work without a key; a missing one is an error.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object EmbeddingConfigSpec

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Stable
trait EmbeddingProviderDescriptor

Everything llm4s needs to know about one embedding provider, supplied by the provider itself.

Everything llm4s needs to know about one embedding provider, supplied by the provider itself.

This is ProviderDescriptor's counterpart for the embedding half of the SPI, and it is deliberately a separate trait rather than a method on the chat descriptor: the two sets overlap but neither contains the other. OpenAI and Ollama supply both a chat client and an embedding provider, Voyage supplies only embeddings, and Anthropic only chat. Folding embeddings into ProviderDescriptor would force an embedding-only provider to implement buildClient and buildConfig only to fail them.

Register it by listing it in an Llm4sProviderModule's Llm4sProviderModule.embeddingProviders, exactly as a chat provider is registered, and EmbeddingClient.from can reach it with nothing in llm4s-core edited.

The module's reference.conf binds the vendor's variable to the shared key, which a section without an apiKey of its own falls back to:

llm4s.credentials.jina.apiKey = ${?JINA_API_KEY}

Attributes

Example 
object JinaEmbeddings extends EmbeddingProviderDescriptor:
 val id = ProviderId("jina")
 override val configSpec = EmbeddingConfigSpec(
   requiresApiKey = true,
   defaultBaseUrl = Some("https://api.jina.ai/v1"),
   apiKeyEnv      = Seq("JINA_API_KEY")   // what llm4s.credentials.jina.apiKey binds
 )
 def build(config: EmbeddingProviderConfig): Result[EmbeddingProvider] =
   Right(JinaEmbeddingProvider.fromConfig(config))
Supertypes
class Object
trait Matchable
class Any
Known subtypes
@Stable
final case class EmbeddingProviderSection(apiKey: Option[String], baseUrl: Option[String], model: Option[String], timeouts: ProviderTimeouts)

One embedding provider's llm4s.embeddings.<id> section, as read from config.

One embedding provider's llm4s.embeddings.<id> section, as read from config.

The shape is the same for every provider, which is what lets llm4s-core parse it without knowing which providers exist: the provider-specific part is what EmbeddingProviderDescriptor.buildConfig makes of it.

Value parameters

apiKey

llm4s.embeddings.<id>.apiKey.

baseUrl

llm4s.embeddings.<id>.baseUrl.

model

llm4s.embeddings.<id>.model.

timeouts

llm4s.embeddings.<id>.timeouts: how long an embedding request may take, as the same timeouts block a chat section has. Only request applies, since an embedding call does not stream. Absent means the provider's own default.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Stable
trait Llm4sProviderModule

A unit of provider registration: everything one module contributes.

A unit of provider registration: everything one module contributes.

The discovered unit is a module rather than a single ProviderDescriptor so that one artifact can supply several related providers — llm4s-openai contributes OpenAI, Azure, Requesty and OpenAI embeddings from a single entry.

Implementations must be a plain class with a public no-arg constructor, '''not''' a Scala object: java.util.ServiceLoader instantiates the named class, and an object exposes its instance as a MODULE$ field instead. Delegate to an object from the class if you want one:

class MyProviderModule extends Llm4sProviderModule:
 override def chatProviders: Seq[ProviderDescriptor]                = Seq(MyProvider)
 override def embeddingProviders: Seq[EmbeddingProviderDescriptor]  = Seq(MyEmbeddings)

Declare the class in META-INF/services/org.llm4s.llmconnect.spi.Llm4sProviderModule and ProviderRegistry.discover finds it - adding the provider is then adding the dependency, with no registration code at the call site. A module can also be registered explicitly through ProviderRegistry.ofModules or ProviderRegistry.withModule, which is the answer for a shaded fat jar whose services files did not survive.

Attributes

Supertypes
class Object
trait Matchable
class Any
Known subtypes
Show all
@Stable
final case class ProviderConfigKey(name: String, description: String, required: Boolean, default: Option[String], env: Option[String], deprecatedAliases: Seq[String])

A provider-specific key in a named provider section: one the fixed fields of NamedProviderConfig do not cover.

A provider-specific key in a named provider section: one the fixed fields of NamedProviderConfig do not cover.

Declaring the key is what makes it part of the section. The section validator checks required keys, fills in defaults and resolves deprecated aliases before the descriptor sees the section, and the descriptor reads the result from NamedProviderConfig.extras. A key in a section that no descriptor declares is reported as unknown and not passed on.

Values are strings, as they are in HOCON; a descriptor that needs a number or a boolean parses it in buildConfig and reports a malformed value there.

Value parameters

default

value used when the section omits the key. A key with a default is never missing, so required is then moot.

deprecatedAliases

older names for this key. A section that sets an alias instead of name still works, with a deprecation warning per alias used; one that sets name and an alias, or two aliases, to different values is an error. An alias may be any key the section could otherwise carry as an extra - which is how a provider renames one (Vertex AI's endpoint became project) - or one of the built-in fields in ProviderConfigSpec.BuiltinAliasKeys. provider, model and headers cannot be aliases, and a spec naming one fails validation.

description

what the key means, shown when a required one is missing - e.g. "the GCP project ID that owns your Vertex AI resources".

env

the conventional environment variable for this key, if it has one. Named sections read no variable by themselves, so the missing-key message shows the binding that reads it - key = ${?VAR} - rather than telling the user to set a variable nothing would read.

name

the key as written in llm4s.providers.<name>, e.g. "region". Must not be one of ProviderConfigSpec.BuiltinKeys.

required

a section without this key (and without a default) is invalid.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object ProviderConfigKey

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Stable
final case class ProviderConfigSpec

The shape of a provider's llm4s.providers.<name> section: which fields it requires, which provider-specific keys it accepts, and what to tell the user when one is missing.

The shape of a provider's llm4s.providers.<name> section: which fields it requires, which provider-specific keys it accepts, and what to tell the user when one is missing.

This replaces the per-provider NamedProviderValidator implementations that llm4s-core used to hold, one object per provider in a shared file. A provider now declares its own requirements, and a single generic validator turns that declaration into the same error messages as before.

== Why defaults live here rather than in HOCON == Chat-provider config is keyed by the user's ''instance'' name (llm4s.providers.my-openai.baseUrl), so a reference.conf fragment shipped by a provider module cannot express "the default baseUrl for anything whose provider = "openai"" — it does not know the instance name. defaultBaseUrl is therefore code, and is the single source for a provider's default endpoint. The same holds for the defaults of extras.

== Provider-specific keys == The built-in fields (ProviderConfigSpec.BuiltinKeys) are shared by every provider. Anything else a provider needs - a cloud region, a project id - is declared in extras rather than smuggled through a built-in field (#1215):

val configSpec = ProviderConfigSpec(
 extras = Seq(
   ProviderConfigKey.required("project", "the GCP project ID that owns your Vertex AI resources"),
   ProviderConfigKey.optional("location", "the GCP region", default = Some("us-central1"))
 )
)

Value parameters

apiKeyEnv

the variables the provider's module binds to its vendor's shared key, llm4s.credentials.<id>.apiKey, highest precedence first - e.g. Seq("OPENAI_API_KEY"), or Gemini's Seq("GOOGLE_API_KEY", "GEMINI_API_KEY"). Named in the missing-apiKey error. This is a declaration for the message only: the binding itself is a line in the module's reference.conf, and the module's round-trip spec proves the two agree. Empty when the module binds no variable; the error then names llm4s.credentials.<id>.apiKey itself.

authExtras

the keys this provider reads from a section's auth block besides the identity token (identityTokenFile / identityToken, which core reads itself). Non-empty means the provider supports workload-identity authentication, and a section with an auth block then needs no apiKey; required and default keys are validated as for extras. == The API key == A section's key is its own apiKey when it sets one, and otherwise the vendor's shared llm4s.credentials.<id>.apiKey - keyed by provider id, which a reference.conf ''can'' know, unlike the instance name. So with OPENAI_API_KEY set, a section needs only provider and model; a section for a second account sets apiKey itself.

baseUrlEnv

the conventional environment variable for baseUrl, if the provider has one. The missing-baseUrl message then shows the binding that reads it (baseUrl = ${?VAR}); None means no variable is suggested.

baseUrlExample

example shown when a required baseUrl is missing.

defaultBaseUrl

base URL used when the section omits one.

extras

the provider-specific keys this section accepts.

requiresApiKey

the section must carry a non-empty apiKey.

requiresBaseUrl

the section must carry a non-empty baseUrl; set this only when there is no defaultBaseUrl.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object ProviderConfigSpec

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Stable
trait ProviderDescriptor

Everything llm4s needs to know about one LLM provider, supplied by the provider itself.

Everything llm4s needs to know about one LLM provider, supplied by the provider itself.

A descriptor is the whole extension point: implement it, register it (by listing it in an Llm4sProviderModule, or by handing it to ProviderRegistry.of), and the provider becomes reachable from configuration, validation, client construction and model discovery without editing anything in llm4s-core. Before this existed, adding a provider meant edits to roughly eight shared files — see #1131.

Attributes

Example 
object BedrockProvider extends ProviderDescriptor:
 val id         = ProviderId("bedrock")
 val configSpec = ProviderConfigSpec(
   extras = Seq(
     ProviderConfigKey.required("region", "the AWS region hosting the model, e.g. us-east-1"),
     ProviderConfigKey.optional("profile", "the AWS named profile to authenticate with")
   )
 )
 def buildConfig(providerName: String, section: NamedProviderConfig)(using
   ContextWindowResolver
 ): Result[ProviderConfig] =
   ProviderDescriptor
     .requireExtra(providerName, section, "region")
     .flatMap(region => BedrockConfig.fromValues(section.model.asString, region, section.extra("profile")))
 def buildClient(config: ProviderConfig, options: LlmClientOptions)(using
   ModelRegistryService
 ): Result[LLMClient] =
   ProviderDescriptor.expectConfig[BedrockConfig](id, config).flatMap(BedrockClient(_, options.metrics))
Companion
object
Supertypes
class Object
trait Matchable
class Any
Known subtypes
object AzureProvider
object ZaiProvider
Show all
object ProviderDescriptor

Attributes

Companion
trait
Supertypes
class Object
trait Matchable
class Any
Self type
@Stable
final case class ProviderDiscoveryFailure(detail: String, failure: Option[Throwable])

A service entry that discovery could not use.

A service entry that discovery could not use.

Recording these rather than letting them propagate is the point of the hand-rolled iteration in ProviderRegistry.discover: one broken jar must not take out every other provider on the classpath.

Value parameters

detail

what went wrong, including the offending class name when java.util.ServiceLoader reported one.

failure

the throwable, kept for logging rather than for control flow.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Stable
final case class ProviderFeatures

What a provider implementation can actually do, declared by the provider itself.

What a provider implementation can actually do, declared by the provider itself.

This is a '''static''' declaration made by a ProviderDescriptor, not a runtime discovery: it describes the client this build ships, so a caller can ask "will streaming work here?" without making a call and reading the error.

The flags default to true because a provider that omits them is claiming the full interface; a provider that cannot honour part of it must say so, and saying so is then visible to users rather than buried in a Left at call time. Cohere and Mistral declared streaming = false until they moved onto llm4s-openai-compatible's shared client, which streams (#925).

Value parameters

streaming

whether LLMClient.streamComplete is implemented.

toolCalling

whether the provider accepts tool/function definitions.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object ProviderFeatures

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
@Stable
final case class ProviderIdCollision(kind: String, id: String, keptModule: String, droppedModule: Option[String])

A chat or embedding provider id two descriptors both registered; the later one won.

A chat or embedding provider id two descriptors both registered; the later one won.

ProviderRegistry has always resolved this by keeping the last registration, silently - this is what makes that resolution visible instead of invisible.

Value parameters

droppedModule

the module that lost, when discovery can name it.

id

the id two descriptors both claimed.

keptModule

the module (or "explicit registration") whose descriptor won.

kind

"chat" or "embedding".

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Stable
final case class ProviderModuleReport(moduleClass: String, providerIds: Seq[String], source: Option[String], embeddingProviderIds: Seq[String])

One Llm4sProviderModule that classpath discovery loaded.

One Llm4sProviderModule that classpath discovery loaded.

Value parameters

embeddingProviderIds

the embedding provider ids it contributed.

moduleClass

fully-qualified name of the module class.

providerIds

the chat provider ids it contributed, in canonical spelling.

source

where the class was loaded from (a jar URL or a directory), when the JVM can say. None for a class with no code source, which is normal under some class loaders.

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
@Stable
final class ProviderRegistry

The set of providers this build can resolve.

The set of providers this build can resolve.

A registry is an immutable value, not a mutable global: withProvider and withModule return a new registry, and an application that wants a different set constructs one with ProviderRegistry.of and passes it in. Core code resolves a registry through a using clause, so the default is ProviderRegistry.default — every provider on the classpath — unless the caller supplies another.

llm4s-core ships no provider: every one comes from a provider module (llm4s-openai, llm4s-anthropic, llm4s-gemini, llm4s-ollama, llm4s-openai-compatible, llm4s-voyage, or a third party's), found through its META-INF/services entry. Where discovery cannot run - a shaded fat jar whose services files were merged away - ProviderRegistry.ofModules builds the registry from the modules named explicitly.

Chat and embedding providers are held in separate namespaces, because the two sets overlap without either containing the other - OpenAI and Ollama supply both, Voyage only embeddings, Anthropic only chat. The same id may therefore appear in both, and does: ollama names a chat client and an embedding provider that share nothing but a base URL.

Lookup is by ProviderId and never throws: an id nothing handles produces a Left naming the ids that are registered, which is the difference between a usable error and "provider openai not registered".

Value parameters

descriptors

the registered chat providers, in registration order.

embeddingDescriptors

the registered embedding providers, in registration order.

report

how this registry came to hold them; see ProviderRegistryReport.

Attributes

Companion
object
Supertypes
class Object
trait Matchable
class Any
object ProviderRegistry

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
@Stable
final case class ProviderRegistryReport(discovered: Boolean, modules: Seq[ProviderModuleReport], failures: Seq[ProviderDiscoveryFailure], collisions: Seq[ProviderIdCollision])

How a ProviderRegistry came to hold what it holds.

How a ProviderRegistry came to hold what it holds.

This is the debuggability half of classpath discovery. "Provider openai is not registered" is unusable on its own; the same message with "discovery scanned 0 modules" says the services files did not survive shading, and with "1 failed: ..." says which jar is broken.

Value parameters

collisions

the ids two descriptors both registered.

discovered

whether ServiceLoader discovery ran at all. A registry built by ProviderRegistry.of reports false, which is different from discovery running and finding nothing.

failures

the service entries it could not use.

modules

the modules discovery loaded, in the order it saw them.

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object ProviderRegistryReport

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
Generated with