Configuration Guide

Configure LLM4S for multiple providers, environments, and use cases.

Table of contents

  1. How configuration is loaded
  2. Named provider sections
    1. API keys
    2. Rotating keys: apiKeyFile and ApiKeyProvider
    3. Workload identity (SPIFFE)
    4. Section keys
    5. Timeouts
    6. Examples for other providers
  3. Corporate networks: a proxy and a custom CA
    1. Your own HTTP client
  4. Switching providers
  5. Using a config your application already has
  6. Running the samples
  7. Embeddings Configuration
    1. Unified Format (Recommended)
    2. Available Models
    3. Legacy Format (Still Supported)
    4. Using Embeddings in Code
    5. Queries and documents
    6. System Properties (Alternative)
  8. Tracing Configuration
    1. Console Tracing (Development)
    2. Langfuse (Production)
    3. OpenTelemetry
    4. Other tracing backends
    5. Disable Tracing
  9. Environment variables llm4s reads
  10. Environment-specific configuration
  11. Configuration Best Practices
    1. ✅ DO
    2. ❌ DON’T
  12. Validation Example
  13. Troubleshooting
    1. Problem: “missing required fields: apiKey”
    2. Problem: “Configured provider ‘…’ was not found”
    3. Problem: “Unknown provider” / provider not registered
    4. Problem: “Configuration not loading from application.conf”
    5. Problem: “Provider mismatch”

How configuration is loaded

Llm4sConfig reads Typesafe Config through PureConfig, with this precedence (highest first):

  1. JVM system properties - -Dllm4s.providers.provider=claude
  2. Your application.conf - src/main/resources/application.conf, or a file named with -Dconfig.file=/path/to/app.conf (or -Dconfig.resource=prod.conf for another resource)
  3. reference.conf - the defaults shipped in each llm4s module on the classpath

Environment variables are not a layer of their own. llm4s reads one only where a ${?VAR} substitution names it: in a module’s reference.conf (tracing, embeddings, tools, and each provider module’s vendor API key - see Environment variables llm4s reads), or in your own application.conf. Each provider module binds its vendor’s conventional key variable - llm4s-openai binds OPENAI_API_KEY, llm4s-anthropic binds ANTHROPIC_API_KEY - to a shared key under llm4s.credentials (see API keys). Nothing reads LLM_MODEL.

${?VAR} means “the value of VAR if it is set, otherwise leave the key unset”. A second line for the same key overrides the first only when the variable is set, which gives you a default with an environment override:

1
2

model = "gpt-4o-mini"
model = ${?OPENAI_MODEL}   # used only when OPENAI_MODEL is set

Named provider sections

Chat providers are configured as named sections under llm4s.providers. You choose each section’s name; provider inside it says which provider it is, and llm4s.providers.provider names the section Llm4sConfig.defaultProvider() loads.

1. Add llm4s and the module for your provider:

1
2
3
4
5

// build.sbt
libraryDependencies ++= Seq(
  "org.llm4s" %% "llm4s-core"   % llm4sVersion,
  "org.llm4s" %% "llm4s-openai" % llm4sVersion  // the module for your provider
)
provider = ... Module
openai, azure, requesty llm4s-openai
anthropic llm4s-anthropic
gemini, vertexai llm4s-gemini
ollama llm4s-ollama
bedrock llm4s-bedrock
deepseek, zai, openrouter, mistral, cohere, xai, openai-compatible llm4s-openai-compatible

Before 0.5.0, every provider shipped inside llm4s-core, so the llm4s-core dependency alone was enough.

2. Add a section to src/main/resources/application.conf:

1
2
3
4
5
6
7
8
9
10

llm4s {
  providers {
    provider = "openai-main"          # the default: the name of a section below

    openai-main {
      provider = "openai"
      model    = "gpt-4o-mini"
    }
  }
}

3. Set your vendor’s API-key variable:

1

export OPENAI_API_KEY=sk-...

The section needs no apiKey line: llm4s-openai binds OPENAI_API_KEY to OpenAI’s shared key, llm4s.credentials.openai.apiKey, and a section without a key of its own uses that one. See API keys for a second account or a variable of your own.

4. Load it at the edge of your application:

1
2
3
4
5
6
7
8
9
10
11
12

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.llmconnect.model.{ Conversation, UserMessage }
import org.llm4s.model.ModelRegistryService

val result = for {
  providerConfig <- Llm4sConfig.defaultProvider()
  registry       <- Llm4sConfig.modelRegistryService()
  given ModelRegistryService = registry
  client   <- LLMConnect.getClient(providerConfig)
  response <- client.complete(Conversation(Seq(UserMessage("What is Scala?"))))
} yield response.content

If OPENAI_API_KEY is unset, defaultProvider() returns a ConfigurationError naming both places a key can come from: apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf. This exact configuration is exercised by DocumentedProviderConfigSpec in modules/openai.

Other Llm4sConfig calls read the same sections:

1
2
3
4
5
6

Llm4sConfig.provider("openai-main")   // any section, by name
Llm4sConfig.defaultProviderName()     // the value of llm4s.providers.provider
Llm4sConfig.providers()               // every section, validated - fails if any one is invalid
Llm4sConfig.providerConfigs()         // every section, each loaded or failed on its own
Llm4sConfig.listModels()              // model discovery for the default section
Llm4sConfig.listModels("openai-main") // ... or for a named one

API keys

Credentials belong to a vendor; sections belong to a use. A client’s key is, in order:

  1. its own apiKey - in its chat section llm4s.providers.<name>, its embeddings block llm4s.embeddings.<provider> or the reranker block llm4s.rerank.cohere;
  2. otherwise its vendor’s shared key, llm4s.credentials.<provider>.apiKey, which the provider module binds to the vendor’s conventional variable;
  3. otherwise a ConfigurationError naming both.

An alias resolves to its canonical provider’s key: a provider = "google" section uses llm4s.credentials.gemini. When a key is resolved, llm4s logs at INFO where it came from - for example llm4s.providers.openai-main: API key from llm4s.credentials.openai.apiKey - and never the value.

Provider id Shared key bound to Module
openai OPENAI_API_KEY llm4s-openai
azure AZURE_OPENAI_API_KEY llm4s-openai
requesty REQUESTY_API_KEY llm4s-openai
anthropic ANTHROPIC_API_KEY llm4s-anthropic
gemini (alias google) GOOGLE_API_KEY, else GEMINI_API_KEY llm4s-gemini
deepseek DEEPSEEK_API_KEY llm4s-openai-compatible
zai ZAI_API_KEY llm4s-openai-compatible
openrouter OPENROUTER_API_KEY llm4s-openai-compatible
mistral MISTRAL_API_KEY llm4s-openai-compatible
xai XAI_API_KEY llm4s-openai-compatible
perplexity PERPLEXITY_API_KEY llm4s-perplexity
cohere COHERE_API_KEY (chat, reranker and embeddings) llm4s-openai-compatible, llm4s-rag, llm4s-cohere
voyage VOYAGE_API_KEY llm4s-voyage
jina JINA_API_KEY llm4s-jina
typesafe TYPESAFE_API_KEY (the Jev decision client; not a chat or embeddings provider) llm4s-jev

ollama takes no key, the generic openai-compatible provider has no vendor, vertexai authenticates with OAuth2 (Application Default Credentials, or a service-account file named by its apiKey), and bedrock authenticates with AWS credentials (the AWS default credential chain, a profile, or explicit keys), so none of them has a shared key.

A section for a second account sets its own key, which wins over the shared one:

1
2
3
4
5
6
7
8
9
10
11
12

llm4s.providers {
  openai-main {                          # uses OPENAI_API_KEY
    provider = "openai"
    model    = "gpt-4o-mini"
  }

  openai-batch {                         # billed to another account
    provider = "openai"
    model    = "gpt-4o-mini"
    apiKey   = ${?OPENAI_BATCH_API_KEY}
  }
}

The same form reads a variable of your own choosing - apiKey = ${?MY_TEAM_OPENAI_KEY}. Note that if OPENAI_BATCH_API_KEY is unset, openai-batch falls back to the shared OPENAI_API_KEY and bills the default account. In production, give every section its own apiKey; the config-policy prod preset flags any section that does not (see Production deployment).

Rotating keys: apiKeyFile and ApiKeyProvider

A key that a secret manager rotates - mounted as a file by Vault Agent, the Secrets Store CSI driver or a Kubernetes secret volume - goes in apiKeyFile instead of apiKey:

1
2
3
4
5

llm4s.providers.openai-main {
  provider   = "openai"
  model      = "gpt-4o-mini"
  apiKeyFile = "/var/run/secrets/llm4s/openai-api-key"
}

The file holds the key alone (surrounding whitespace is trimmed). It is read when the section is loaded - a missing, unreadable or empty file fails the load with a ConfigurationError naming the path - and the client checks it again before every request (one stat), re-reading it whenever its modification time, size or identity changes. A key rewritten in place, or swapped in through a symbolic link as Kubernetes does, is sent from the next request on, without rebuilding the client or restarting the application. The INFO line names the file, never the key.

A section sets apiKeyFile or apiKey (or auth), never two of them, and with apiKeyFile the vendor’s shared key is not used. Every built-in provider that takes an API key re-reads the file per request as above: openai, requesty, azure (its api-key header), anthropic, gemini, openai-compatible, openrouter, deepseek, zai, mistral, cohere, xai, perplexity, and watsonx, which exchanges a rotated key for a new IAM token at the next request. A provider from another module whose config cannot take an ApiKeyProvider reads the file once, when the section is loaded, and logs that it does. A provider that takes no API key (ollama, bedrock, vertexai) refuses apiKeyFile with a ConfigurationError. On a POSIX file system, a key file readable by its group or by others is logged once as a WARN naming the path; mount it with mode 0400.

apiKeyFile is accepted wherever apiKey is: in an embeddings block (llm4s.embeddings.<id>), in the reranker’s block (llm4s.rerank.cohere) and in the vendor’s shared credentials (llm4s.credentials.<id>), which every client of the vendor falls back to. The precedence is the same as for apiKey: the client’s own key or key file, then the shared key or key file, then an error naming both. So one mounted file can serve a vendor’s chat sections, its embeddings and the reranker:

1
2
3
4
5

llm4s {
  credentials.openai.apiKeyFile = "/var/run/secrets/llm4s/openai-api-key"   # every OpenAI client
  embeddings.voyage.apiKeyFile  = "/var/run/secrets/llm4s/voyage-api-key"   # Voyage embeddings only
  rerank.cohere.apiKeyFile      = "/var/run/secrets/llm4s/cohere-api-key"   # the Cohere reranker only
}

The embedding providers (openai, voyage, jina, cohere, ollama) and the Cohere reranker re-read the file per request, as the chat clients above do; a chat section that falls back to a shared apiKeyFile is treated as if it set that file itself. A block or credentials entry that sets both apiKey and apiKeyFile is refused - a provider module’s reference.conf binds llm4s.credentials.<id>.apiKey to the vendor’s variable, so with a shared apiKeyFile, leave that variable unset. The safety rules are the same: the path is logged, never the key; a file over 64 KiB is refused; a group- or world-readable file is logged as a WARN.

A key your code fetches - from Vault’s API, AWS Secrets Manager or GCP Secret Manager - comes from an ApiKeyProvider, which the client asks for the key on every request. current() is on the request path, so it caches:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

import org.llm4s.config.Llm4sConfig
import org.llm4s.error.ConfigurationError
import org.llm4s.llmconnect.{ LLMClient, LLMConnect }
import org.llm4s.llmconnect.auth.ApiKeyProvider
import org.llm4s.model.ModelRegistryService
import org.llm4s.types.ProviderModelTypes.ApiKey
import org.llm4s.types.Result

import java.time.Instant
import java.util.concurrent.atomic.AtomicReference
import scala.concurrent.duration.*

/** The key `fetch` returns, fetched again once it is older than `maxAge`. */
class SecretStoreKey(fetch: () => Result[String], maxAge: FiniteDuration) extends ApiKeyProvider:
  private val cached = AtomicReference(Option.empty[(ApiKey, Instant)])

  def current(): Result[ApiKey] =
    cached.get() match
      case Some((key, fetchedAt)) if Instant.now().isBefore(fetchedAt.plusMillis(maxAge.toMillis)) => Right(key)
      case _ =>
        fetch().map { value =>
          val key = ApiKey(value.trim)
          cached.set(Some(key -> Instant.now()))
          key
        }

// Your secret store's client goes here - for AWS Secrets Manager, `GetSecretValue` on the secret's id.
val fetchOpenAIKey: () => Result[String] = () => Left(ConfigurationError("connect the secret store"))

val rotating: Result[LLMClient] = for {
  providerConfig <- Llm4sConfig.provider("openai-main")
  registry       <- Llm4sConfig.modelRegistryService()
  given ModelRegistryService = registry
  config = providerConfig.withApiKeyProvider(SecretStoreKey(fetchOpenAIKey, 5.minutes))
  // A provider whose client cannot take one leaves `apiKeyProvider` empty.
  _      <- Either.cond(config.apiKeyProvider.isDefined, (), ConfigurationError("this provider cannot rotate keys"))
  client <- LLMConnect.getClient(config)
} yield client

ApiKeyProvider.file(path) is the provider apiKeyFile sets, and ApiKeyProvider.static(key) one that never changes. A key created as an ApiKey is registered for redaction, so an error that echoes a rotated key is scrubbed of it.

Workload identity (SPIFFE)

A section can authenticate with a workload identity token - typically a SPIFFE JWT-SVID that spiffe-helper keeps fresh in a file - instead of an API key. Put an auth block in the section, with identityTokenFile (re-read on every exchange, so rotation is picked up) and the keys the provider needs. A relative identityTokenFile is resolved against the process’s working directory when the configuration is loaded, so prefer an absolute path. The path must name a regular file of at most 64 KiB (a symbolic link to one is followed, as Kubernetes projected volumes rotate tokens that way); anything else fails the exchange with an AuthenticationError that names the path but never quotes the file. For openai-compatible and openai llm4s reads the file itself; for anthropic the Anthropic SDK reads it. Keep the file readable only by the workload where the platform allows it. A section sets apiKey or auth, never both, and a section with auth may not set an Authorization header either: the exchanged token is the request’s bearer. With auth, the shared llm4s.credentials.<id>.apiKey is not used.

Databricks model serving (the generic openai-compatible provider; the SVID is exchanged at the workspace’s RFC 8693 endpoint, the token cached until shortly before it expires, and refreshed once if a request is rejected with 401):

1
2
3
4
5
6
7
8
9
10
11

databricks-main {
  provider = "openai-compatible"
  baseUrl  = "https://<workspace>.cloud.databricks.com/serving-endpoints"
  model    = "<serving endpoint name>"
  auth {
    identityTokenFile = "/var/run/secrets/spiffe/databricks"
    tokenUrl = "https://<workspace>.cloud.databricks.com/oidc/v1/token"
    clientId = ${?DATABRICKS_CLIENT_ID}   # the service principal, for a service-principal federation policy
    scope    = "all-apis"
  }
}

The optional audience key sets the RFC 8693 audience parameter, for a token endpoint that needs one. tokenUrl and baseUrl must be https (plain http only for a loopback host). The exchanged token is sent as the bearer of every request to baseUrl, a host you choose, so no host is allow-listed: make sure tokenUrl and baseUrl belong to the same trusted service. The token is cached for the reply’s expires_in; a reply without one is taken to live until the access token’s exp claim if it is a JWT, and otherwise for 5 minutes.

OpenAI (the OpenAI SDK’s workload identity federation):

1
2
3
4
5
6
7
8
9

openai-wif {
  provider = "openai"
  model    = "gpt-4o-mini"
  auth {
    identityTokenFile  = "/var/run/secrets/spiffe/openai"
    identityProviderId = ${?OPENAI_IDENTITY_PROVIDER_ID}
    serviceAccountId   = ${?OPENAI_SERVICE_ACCOUNT_ID}
  }
}

Anthropic (the Anthropic SDK’s workload identity federation; identityTokenFile only, not a literal identityToken):

1
2
3
4
5
6
7
8
9
10
11

anthropic-wif {
  provider = "anthropic"
  model    = "claude-sonnet-4-5"
  auth {
    identityTokenFile = "/var/run/secrets/spiffe/anthropic"
    federationRuleId  = ${?ANTHROPIC_FEDERATION_RULE_ID}
    organizationId    = ${?ANTHROPIC_ORGANIZATION_ID}
    serviceAccountId  = ${?ANTHROPIC_SERVICE_ACCOUNT_ID}
    workspaceId       = ${?ANTHROPIC_WORKSPACE_ID}
  }
}

A ${?VAR} that is unset leaves its key out, and a missing required key is reported when the section is loaded. With auth, baseUrl must be the vendor’s own API host over https, because the exchanged token is a vendor credential sent with every request (plain http is accepted only for a loopback host, for tests): for OpenAI api.openai.com or a data-residency host (us., eu. or ae.api.openai.com), for Anthropic api.anthropic.com, where the SDK also posts the identity token (<baseUrl>/v1/oauth/token). A proxy or gateway baseUrl needs an apiKey instead.

The Anthropic SDK reads identityTokenFile itself, so llm4s’s own checks on a token file it reads (a regular file of at most 64 KiB, read with a bounded read) do not guard the SDK’s reads. Building the client applies them once: a path that is a directory, a FIFO, a device or a file larger than 64 KiB is refused then, as a ConfigurationError naming workloadIdentity.identityTokenFile. A path that does not exist yet is accepted, because an identity agent may write it after the application starts. Point it at the file the agent keeps fresh.

Request each SVID for the audience its relying party expects (jwt_audience in spiffe-helper). Other providers reject an auth block. The prod config-policy preset’s ownApiKey rule accepts a section that authenticates this way.

Model listing works for an openai-compatible section with auth (it exchanges once). For openai and anthropic sections with auth it is not supported - their SDKs perform the exchange only inside a client - and the lister returns a ConfigurationError saying so; list models from a section with an apiKey.

Section keys

Key Meaning
provider Required. The provider id, e.g. openai, anthropic, ollama (table above)
model Required. The model name as the provider spells it, e.g. gpt-4o-mini
apiKey The API key; required by every cloud provider. Optional in the section when the vendor’s shared key is set (API keys)
apiKeyFile A file holding the API key, instead of apiKey. Re-read when it changes (no restart) by every built-in provider that takes a key (Rotating keys)
baseUrl Overrides the provider’s default endpoint (ollama: http://localhost:11434); required for openai-compatible. Checked when the section is loaded: an absolute http or https URL with a host, or a ConfigurationError naming the key (a blank value counts as unset)
headers Extra HTTP headers; sent by generic openai-compatible endpoints and by model listing
timeouts How long a request and a stream may take: timeouts { request = 3m, stream = 15m }. Every provider that makes HTTP calls reads it; see Timeouts
organization OpenAI, Requesty and OpenRouter: the OpenAI organisation id, sent as OpenAI-Organization
endpoint, apiVersion Azure OpenAI: the resource endpoint (required) and API version
project, location Vertex AI: the GCP project id (required) and region (default us-central1)
region, profile, accessKeyId, secretAccessKey, sessionToken AWS Bedrock: the AWS region (required, never defaulted); a shared-config profile; or explicit credentials, with a session token for temporary ones. With none of the credential keys the AWS default credential chain is used. baseUrl overrides the endpoint
contextWindow, reserveCompletion, registryProvider, streamUsage Generic openai-compatible endpoints (details)

The first seven keys are shared by every provider (apiKeyFile by every one that takes an API key). The rest belong to the providers named, which declare them (provider-specific keys); in a section for any other provider such a key is ignored with a warning naming it, as a misspelt key is.

Each provider module’s reference.conf has a commented example section, and the provider guide covers each provider in detail.

Timeouts

Every provider section, and every embedding section, accepts an optional timeouts block:

1
2
3
4
5
6
7
8
9
10

llm4s.providers.my-openai {
  provider = "openai"
  model    = "gpt-4o"
  timeouts {
    request = 3m      # a call that returns one response: a completion, an embedding
    stream  = 15m     # a streamed completion
  }
}

llm4s.embeddings.openai.timeouts.request = 30s

Each value is a duration (30s, 2m, 1500ms, 1h) and is optional: a value you leave out keeps that client’s own default, so a section without the block behaves exactly as it did before. A value must be positive and finite; 0s, a negative value, Inf and anything that is not a duration are refused when the section is loaded, naming the key (llm4s.providers.my-openai.timeouts.request). A misspelt key inside the block (reqest = 3m) is refused too, rather than silently leaving the default in force. In an embedding section only request has a meaning, since an embedding call does not stream. The usual precedence applies: -D system properties, then application.conf, then a module’s reference.conf; to take a value from the environment, bind it yourself with timeouts.request = ${?MY_REQUEST_TIMEOUT}, which leaves the default in force while the variable is unset.

Provider request default stream default What the timeout bounds
openai-compatible, deepseek, zai, openrouter, mistral, cohere (chat) 2 minutes 5 minutes the wait for the response to begin
gemini, vertexai, ollama, watsonx (chat) 2 minutes 10 minutes the wait for the response to begin
perplexity 5 minutes 10 minutes the wait for the response to begin
openai, azure, requesty, anthropic the SDK’s own default the SDK’s own default the whole call, including a streamed body
bedrock the AWS SDK’s own default no limit the whole call, including retries and a streamed body
Embeddings: openai, ollama 2 minutes n/a the wait for the response to begin
Embeddings: voyage, jina, cohere 2 minutes n/a the wait for the response to begin

Five things to know:

  • The HTTP-based clients bound the wait for the response to begin, not the time a stream may then run. A stream whose server has begun answering is not cut by stream.
  • The OpenAI, Azure, Requesty and Anthropic clients use their vendor’s SDK, which bounds the whole call and retries a call that fails or times out twice by default (llm4s does not change that). A request = 30s there can therefore take up to three attempts and their backoff before the call fails. request and stream are independent: a short request does not cut a stream.
  • Bedrock uses the AWS SDK. request becomes the SDK’s API-call timeout, which covers its retries, so a request = 30s call fails within 30 seconds; stream is a deadline on the whole ConverseStream call, so unlike the HTTP-based clients it does cut a stream that is still running. Either expiry is a TimeoutError.
  • The workload identity token exchange of an openai-compatible section with auth (see Workload identity) follows that section’s timeouts.request, so a slow token endpoint is given up on when a slow completion would be; with request unset it uses the request default, 2 minutes.
  • Model listing, the Vertex AI token request and the watsonx IAM token exchange keep their own fixed timeouts, which the block does not change. That includes the exchange model listing makes for an openai-compatible section with auth.

Examples for other providers 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

llm4s {
  providers {
    provider = "claude"

    claude {                              # key from ANTHROPIC_API_KEY
      provider = "anthropic"
      model    = "claude-sonnet-4-20250514"
    }

    gemini-main {                         # key from GOOGLE_API_KEY or GEMINI_API_KEY
      provider = "gemini"
      model    = "gemini-2.0-flash"
    }

    azure-main {                          # key from AZURE_OPENAI_API_KEY
      provider   = "azure"
      model      = "gpt-4o"
      endpoint   = ${?AZURE_API_BASE}      # https://<resource>.openai.azure.com
      apiVersion = ${?AZURE_API_VERSION}   # optional
    }

    ollama-local {
      provider = "ollama"
      model    = "llama3.2"
      baseUrl  = ${?OLLAMA_BASE_URL}       # optional override; the default is http://localhost:11434
    }
  }
}

Only the section you load is validated. A section with no key available, or whose provider module is not on the classpath, fails when it is loaded - by provider("<name>"), or by defaultProvider() when it is the default - and not otherwise. With the example above and ANTHROPIC_API_KEY alone set, defaultProvider() loads claude while provider("gemini-main") fails with a ConfigurationError naming GOOGLE_API_KEY, GEMINI_API_KEY and gemini-main’s apiKey. The exception is Llm4sConfig.providers(), which returns every section and so validates them all; Llm4sConfig.providerConfigs() instead reports each section’s error separately. (Up to 0.4.1 every section was validated on every load.)

Corporate networks: a proxy and a custom CA

Behind an outbound proxy, or behind TLS inspection that re-signs traffic with a corporate CA, the first call fails with connection timed out or PKIX path building failed. One llm4s.http block fixes both for every provider client: the JDK client in core (the OpenAI-compatible providers, Ollama, Gemini, Vertex AI, watsonx.ai, the embedding providers and the model listers), the openai-java SDK (OpenAI, Azure OpenAI), the anthropic-java SDK and the AWS SDK (Bedrock) (#2273).

1
2
3
4
5
6
7
8
9
10
11
12
13
14

llm4s.http {
  proxy {
    url           = "http://proxy.corp:3128"        # bound to HTTPS_PROXY; an http proxy, with its port
    username      = ${?LLM4S_PROXY_USER}            # Basic proxy authentication, if the proxy asks for it
    password      = ${?LLM4S_PROXY_PASSWORD}
    nonProxyHosts = ["localhost", "127.*", "*.internal"]   # bound to NO_PROXY; reached directly
  }
  tls {
    trustStore         = "/etc/pki/corp-ca.p12"     # PKCS12 or JKS; trusted in addition to the JDK's CAs
    trustStorePassword = ${?LLM4S_TRUSTSTORE_PASSWORD}
    keyStore           = "/etc/pki/client.p12"      # optional: a client certificate, for mTLS gateways
    keyStorePassword   = ${?LLM4S_KEYSTORE_PASSWORD}
  }
}
  • Core’s reference.conf binds HTTPS_PROXY, NO_PROXY, LLM4S_PROXY_USER, LLM4S_PROXY_PASSWORD, LLM4S_TRUSTSTORE, LLM4S_TRUSTSTORE_PASSWORD, LLM4S_KEYSTORE and LLM4S_KEYSTORE_PASSWORD, so in a container the variables alone are enough. A user:password@ in the proxy URL is read as the credentials too.
  • nonProxyHosts entries are a host (ollama.local), a domain suffix (*.internal or .internal), a prefix (10.*) or *. The default is the loopback addresses, so a local Ollama is reached directly; setting the list replaces that default. CIDR ranges are not supported.
  • The trust store adds CAs: the public CAs of the JDK stay trusted.
  • An unknown key, a proxy URL that is not http://host:port, or a store that cannot be opened (missing, corrupt, wrong password) fails the load with a ConfigurationError naming the key. Passwords never appear in an error, a log line or a toString.

A section that needs a different route - a gateway reached through another proxy, or directly - sets its own http block, whose keys take precedence over llm4s.http one by one:

1
2
3
4
5
6

llm4s.providers.internal-gateway {
  provider = "openai-compatible"
  model    = "llama-3-70b"
  baseUrl  = "https://llm.internal.corp/v1"
  http.proxy.url = ""                               # no proxy for this section
}

An HTTP client of your own - a tool, a vector store, an MCP transport - takes the same settings through Llm4sHttpClient.create:

1
2
3
4

import org.llm4s.config.Llm4sConfig
import org.llm4s.http.Llm4sHttpClient

val routedClient = Llm4sConfig.httpNetwork().map(network => Llm4sHttpClient.create(network))

Java builders take the same settings with proxy, proxyCredentials, nonProxyHosts, trustStore and keyStore (see the Java guide); the Spring Boot starter binds the llm4s.http.* properties (llm4s.http.proxy.url, llm4s.http.proxy.non-proxy-hosts, llm4s.http.tls.trust-store, …).

What is not covered:

  • PAC files and Kerberos/NTLM proxy authentication. Configure the JVM instead: -Djava.net.useSystemProxies=true for the operating system’s proxy settings, and the JDK’s jdk.http.auth.* properties for the schemes it supports.
  • Basic proxy authentication over an https tunnel in the JDK client. The JDK disables Basic for CONNECT by default; start the JVM with -Djdk.http.auth.tunneling.disabledSchemes="" to allow it. The openai-java, anthropic-java and AWS SDK clients authenticate the tunnel without it.
  • Bedrock’s credential lookups (instance metadata, STS) use the AWS SDK’s own clients and are not routed through llm4s.http; the AWS SDK reads HTTPS_PROXY for those itself.
  • Clients you build yourself with Llm4sHttpClient.create() keep the JVM’s defaults; pass the settings as above.

Your own HTTP client

A service that already has a tuned HTTP stack - connection pool, proxy, request logging, tracing - can hand it to the provider clients instead: implement org.llm4s.http.Llm4sHttpClient over it and pass it with LlmClientOptions.withHttpClient (#2341).

1
2
3
4
5
6
7
8
9
10
11
12
13

import org.llm4s.config.Llm4sConfig
import org.llm4s.http.Llm4sHttpClient
import org.llm4s.llmconnect.{ LLMConnect, LlmClientOptions }
import org.llm4s.model.ModelRegistryService

val http: Llm4sHttpClient = Llm4sHttpClient.create() // or your own implementation of the trait

val client = for {
  providerConfig <- Llm4sConfig.defaultProvider()
  registry       <- Llm4sConfig.modelRegistryService()
  given ModelRegistryService = registry
  client <- LLMConnect.getClient(providerConfig, LlmClientOptions.default.withHttpClient(http))
} yield client
  • You own it. Closing the LLM client does not close a supplied HTTP client; close it when your application stops.
  • The section’s llm4s.http proxy and TLS settings do not apply to it: configure them on your own stack.
  • SDK-free providers use it: ollama, gemini, vertexai, watsonx, openai-compatible and its dialects (deepseek, zai, openrouter, mistral, cohere, xai). Providers built on a vendor SDK (openai, azure, requesty, anthropic, bedrock) keep the SDK’s transport and ignore it. A descriptor’s features.customHttpClient says which: it is false unless the provider opts in, so a third-party provider that predates the option does not claim it. Embedding providers do not take one yet.

Switching providers

Your code calls Llm4sConfig.defaultProvider() and does not change. To use a different provider, point llm4s.providers.provider at another section:

  • Edit the file: provider = "claude".

  • Bind it to a variable of your own:

    1
2
3
4

    llm4s.providers {
  provider = "openai-main"
  provider = ${?LLM4S_PROVIDER}   # your binding; any variable name works
}

    then export LLM4S_PROVIDER=claude.

  • Override it for one run: sbt -Dllm4s.providers.provider=claude run, or java -Dllm4s.providers.provider=claude -jar app.jar.
  • Load a section by name instead of the default: Llm4sConfig.provider("claude"). This is how an application uses several providers at once.

For per-environment setups (local Ollama in development, a cloud provider in production), keep one file per environment - application.conf plus prod.conf that starts with include "application.conf" - and pick one with -Dconfig.resource=prod.conf. This is a matter of taste, not a requirement: a section an environment has no key for does no harm there unless it is loaded.

Using a config your application already has

Llm4sConfig’s methods read the process-wide application.conf. When your application already holds its configuration as a Typesafe Config - Play’s Configuration.underlying, a Pekko ActorSystem.settings.config, one config per tenant, or one parsed per test case - pass it to Llm4sConfig.from. It returns an Llm4sSettings with the same methods (defaultProvider(), provider(name), providers(), embeddings(), tracing(), modelRegistryService(), …):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import com.typesafe.config.{ Config, ConfigFactory }
import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.{ LLMClient, LLMConnect }
import org.llm4s.types.Result

// In a Play or Pekko app this is the Config the framework already loaded.
val appConfig: Config = ConfigFactory.parseString(
  """llm4s.providers {
    |  provider = "openai-main"
    |  openai-main { provider = "openai", model = "gpt-4o-mini" }
    |}""".stripMargin
)

val llmClient: Result[LLMClient] = for {
  settings <- Llm4sConfig.from(appConfig)
  registry <- settings.modelRegistryService()
  cfg      <- settings.defaultProvider()
  client   <- LLMConnect.getClient(cfg)(using registry)
} yield client

The given config takes the place of application.conf and nothing else changes, exactly as ConfigFactory.load(config) would: JVM system properties still override it, every llm4s module’s reference.conf still sits beneath it - so the shared llm4s.credentials.<id>.apiKey fallback (OPENAI_API_KEY above) and each module’s defaults apply - and ${?VAR} substitutions are resolved against the process environment. A config that is already loaded and resolved is accepted as is. Two Llm4sSettings in one JVM never see each other’s configuration. from returns a Left(ConfigurationError) when the config cannot be resolved, such as a ${VAR} substitution with no value.

Llm4sConfig.from and Llm4sSettings are @Experimental: their shape may still change before 1.0.

Running the samples

The repository’s samples read modules/samples/src/main/resources/application.conf, which defines one section, ollama-local, as the default, and binds three variables of its own:

Variable Bound by the samples’ application.conf to
LLM4S_PROVIDER llm4s.providers.provider - which section to use
OLLAMA_MODEL llm4s.providers.ollama-local.model (default llama3:latest)
OLLAMA_BASE_URL llm4s.providers.ollama-local.baseUrl (default http://localhost:11434)

These are the samples’ bindings, not the library’s: your own application reads them only if its application.conf binds them.

With the default section, the model must be in your Ollama install: run ollama pull llama3 once, or name a model you already have (ollama list) with export OLLAMA_MODEL=<model>.

To run a sample against another provider, add a section to modules/samples/src/main/resources/application.local.conf - git-ignored and included by the samples’ application.conf - and select it:

1
2
3
4
5
6
7

# modules/samples/src/main/resources/application.local.conf
llm4s.providers {
  openai-main {
    provider = "openai"
    model    = "gpt-4o-mini"
  }
}

1
2
3

export OPENAI_API_KEY=sk-...
export LLM4S_PROVIDER=openai-main
sbt "samples/runMain org.llm4s.samples.basic.BasicLLMCallingExample"

The chat-tui sample is the exception: ChatTuiConfig reads LLM_MODEL=<provider>/<model> and the matching API-key variable itself, and falls back to Llm4sConfig.defaultProvider() when LLM_MODEL is unset.

Embeddings Configuration

LLM4S supports multiple embedding providers for RAG (Retrieval-Augmented Generation) workflows.

Embeddings are configured separately from chat providers, under llm4s.embeddings. Select the provider and model with llm4s.embeddings.model in provider/model-name form, which core’s reference.conf binds to EMBEDDING_MODEL:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

# OpenAI embeddings (the same OPENAI_API_KEY the chat sections use)
EMBEDDING_MODEL=openai/text-embedding-3-small
OPENAI_API_KEY=sk-...

# Voyage AI embeddings
EMBEDDING_MODEL=voyage/voyage-3
VOYAGE_API_KEY=pa-...

# Jina AI embeddings
EMBEDDING_MODEL=jina/jina-embeddings-v3
JINA_API_KEY=jina_...

# Cohere embeddings
EMBEDDING_MODEL=cohere/embed-english-v3.0
COHERE_API_KEY=...

# Ollama embeddings (local, no API key needed)
EMBEDDING_MODEL=ollama/nomic-embed-text

The key follows the same order as a chat section’s: llm4s.embeddings.<provider>.apiKey if you set it, otherwise the vendor’s shared llm4s.credentials.<provider>.apiKey. So one OPENAI_API_KEY serves both OpenAI chat and OpenAI embeddings. To bill embeddings to another account, give the block its own key:

1

llm4s.embeddings.openai.apiKey = ${?OPENAI_EMBEDDINGS_API_KEY}

Each embedding provider comes from its module: openai from llm4s-openai, voyage from llm4s-voyage, jina from llm4s-jina, cohere from llm4s-cohere and ollama from llm4s-ollama (in 0.4.1 and earlier, openai and voyage are inside llm4s-core). Without the module, the provider id fails with an error naming the registered embedding providers.

Default base URLs are used automatically:

  • OpenAI: https://api.openai.com/v1
  • Voyage: https://api.voyageai.com/v1
  • Jina: https://api.jina.ai/v1
  • Cohere: https://api.cohere.com (the API root; the provider posts to /v2/embed)
  • Ollama: http://localhost:11434

Override base URLs if needed - each module’s reference.conf binds a variable for it:

1
2
3
4
5

OPENAI_EMBEDDING_BASE_URL=https://custom.openai.com/v1   # llm4s-openai
VOYAGE_EMBEDDING_BASE_URL=https://custom.voyage.ai/v1    # llm4s-voyage
JINA_EMBEDDING_BASE_URL=https://custom.jina.ai/v1        # llm4s-jina
COHERE_EMBEDDING_BASE_URL=https://custom.cohere.com      # llm4s-cohere
OLLAMA_EMBEDDING_BASE_URL=http://embeddings-host:11434   # llm4s-ollama

Available Models

OpenAI:

  • text-embedding-3-small - Fast, cost-effective (1536 dimensions)
  • text-embedding-3-large - Higher quality (3072 dimensions)
  • text-embedding-ada-002 - Legacy model (1536 dimensions)

Voyage AI:

  • voyage-3 - General purpose
  • voyage-3-large - Higher quality
  • voyage-code-2 - Code-optimized

Voyage embeds queries and documents differently. Every request is sent with an input_type of document or query, taken from the request’s purpose (see Queries and documents below). Requests that do not say get document.

Jina AI:

  • jina-embeddings-v3 - Multilingual, 8192-token context (1024 dimensions)
  • jina-embeddings-v4 - (2048 dimensions)

Jina embeds queries and documents differently. The provider built from EMBEDDING_MODEL follows each request’s purpose (see Queries and documents below): a document is sent with the retrieval.passage task and a query with retrieval.query. To send one task whatever the request says, for example text-matching, build the provider with the typed JinaTask setting: JinaEmbeddingProvider.fromConfig(config, JinaTask.TextMatching). An explicit task wins over the purpose.

Cohere:

  • embed-v4.0 - (1536 dimensions)
  • embed-english-v3.0, embed-multilingual-v3.0 - (1024 dimensions)
  • embed-english-light-v3.0, embed-multilingual-light-v3.0 - (384 dimensions)

Cohere embeds queries and documents differently. The provider built from EMBEDDING_MODEL follows each request’s purpose (see Queries and documents below): a document is sent with input_type search_document and a query with search_query. To send one input type whatever the request says, for example classification, build the provider with the typed CohereInputType setting: CohereEmbeddingProvider.fromConfig(config, CohereInputType.Classification). An explicit input type wins over the purpose. Texts are sent in requests of at most 96, Cohere’s limit, and the vectors keep the model’s default size.

Ollama (local):

  • nomic-embed-text - General purpose (768 dimensions)
  • mxbai-embed-large - Higher quality (1024 dimensions)
  • all-minilm - Lightweight (384 dimensions)

Install Ollama embedding model:

1

ollama pull nomic-embed-text

Legacy Format (Still Supported)

The legacy format using EMBEDDING_PROVIDER is still supported for backward compatibility:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# OpenAI
EMBEDDING_PROVIDER=openai
OPENAI_EMBEDDING_BASE_URL=https://api.openai.com/v1
OPENAI_EMBEDDING_MODEL=text-embedding-3-small
OPENAI_API_KEY=sk-...

# Voyage AI
EMBEDDING_PROVIDER=voyage
VOYAGE_EMBEDDING_BASE_URL=https://api.voyageai.com/v1
VOYAGE_EMBEDDING_MODEL=voyage-3
VOYAGE_API_KEY=pa-...

# Ollama
EMBEDDING_PROVIDER=ollama
OLLAMA_EMBEDDING_BASE_URL=http://localhost:11434
OLLAMA_EMBEDDING_MODEL=nomic-embed-text

Using Embeddings in Code 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.EmbeddingClient
import org.llm4s.llmconnect.config.EmbeddingModelConfig
import org.llm4s.agent.memory.LLMEmbeddingService
import org.llm4s.model.ModelRegistryService

// Core logic depends on injected service
class RAGService(embeddingService: LLMEmbeddingService) {
  def processDocuments(docs: Seq[String]): Unit = {
    // Use embeddingService...
  }
}

// Configuration boundary
object RAGApplication extends App {
  val startup = for {
    // 1. Load config
    embeddingConfig <- Llm4sConfig.embeddings()
    (provider, cfg) = embeddingConfig
    model    <- Llm4sConfig.textEmbeddingModel()
    registry <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry

    // 2. Build dependencies
    client <- EmbeddingClient.from(provider, cfg)
    
    // 3. Create service
    service = LLMEmbeddingService(
      client, 
      EmbeddingModelConfig(model.modelName, model.dimensions)
    )
  } yield new RAGService(service)

  startup.fold(
    err => println(s"Startup failed: $err"),
    service => println("RAG Service started successfully")
  )
}

Queries and documents

Several embedding models embed a search query differently from a document, so a query only finds what was indexed if the two were embedded on matching sides. An EmbeddingRequest says which it is with its purpose:

1
2
3
4

import org.llm4s.llmconnect.model.{ EmbeddingRequest, InputPurpose }

val indexing  = EmbeddingRequest(chunks, model)                            // a document: the default
val searching = EmbeddingRequest(Seq(question), model, InputPurpose.Query) // a query

Each provider maps the purpose onto its own parameter:

Provider Document Query
Voyage input_type document input_type query
Jina task retrieval.passage task retrieval.query
Cohere input_type search_document input_type search_query
OpenAI, Ollama nothing sent: these models embed both alike nothing sent

RAG and the benchmark RAGPipeline already do this: what they ingest is embedded as documents and the question they answer as a query. Code that does not say keeps embedding documents.

  • An explicit setting wins. A Jina task or a Cohere input type passed to the provider’s fromConfig is sent for every request, whatever its purpose, because it can be one the purpose cannot express (text-matching, classification, clustering).
  • Voyage now sends input_type. Before, it sent none, which Voyage treats as a plain embedding. Documents indexed before this change were embedded that way; they still work against queries, and re-indexing makes the two sides match exactly.
  • Caching keeps the two apart. CachedEmbeddingClient’s key includes the request’s purpose, so a query and a document with the same text never share an entry (see the caching guide).

System Properties (Alternative)

When running via sbt, use system properties for reliable configuration:

1
2
3
4
5
6
7
8

# Using unified format (recommended)
sbt -Dllm4s.embeddings.model=ollama/nomic-embed-text \
    "run"

# Or with legacy format
sbt -Dllm4s.embeddings.provider=ollama \
    -Dllm4s.embeddings.ollama.model=nomic-embed-text \
    "run"

Tracing Configuration

Console Tracing (Development) 

1

TRACING_MODE=console

Console tracing is the default. It prints each event to stdout as metadata only - event type, model, ids, tool names, durations, token counts and costs - and withholds queries, model output, tool input/output, custom event data and error messages (Query: [withheld: 42 chars]). To print those too, while developing:

1

LLM4S_TRACING_CONSOLE_CAPTURE_CONTENT=true   # llm4s.tracing.console.captureContent = true

Keep it off wherever stdout is collected into logs; see Data protection.

Langfuse (Production)

Langfuse tracing needs the llm4s-observability module on the classpath:

1

libraryDependencies += "org.llm4s" %% "llm4s-observability" % "<version>"

1
2
3
4

TRACING_MODE=langfuse
LANGFUSE_PUBLIC_KEY=pk-lf-...
LANGFUSE_SECRET_KEY=sk-lf-...
LANGFUSE_URL=https://cloud.langfuse.com

That module’s reference.conf binds the LANGFUSE_* variables under llm4s.tracing.langfuse, so they can equally be set there in application.conf. Both keys are required: without one, Tracing.fromSettings returns a ConfigurationError naming llm4s.tracing.langfuse.publicKey (LANGFUSE_PUBLIC_KEY) and/or secretKey, and Tracing.create logs it and traces nothing. Without the module, TRACING_MODE=langfuse gives an error naming the llm4s-observability artifact.

Get keys from Langfuse:

  1. Sign up at langfuse.com
  2. Create a project
  3. Navigate to Settings → API Keys
  4. Copy public and secret keys

OpenTelemetry

OpenTelemetry provides distributed tracing for observability across your infrastructure.

Setup:

1
2
3

TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-agent
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

OpenTelemetry tracing needs the llm4s-observability-otel module on the classpath.

Settings (bound in core’s reference.conf):

Config key Variable Default Description
llm4s.tracing.mode TRACING_MODE console Must be opentelemetry
llm4s.tracing.opentelemetry.serviceName OTEL_SERVICE_NAME llm4s-agent Service name for trace identification
llm4s.tracing.opentelemetry.endpoint OTEL_EXPORTER_OTLP_ENDPOINT http://localhost:4317 OpenTelemetry Collector endpoint
llm4s.tracing.opentelemetry.headers (none) (empty) Headers sent with every export
llm4s.tracing.opentelemetry.captureContent OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT false Record prompts and completions, redacted, as gen_ai.input.messages / gen_ai.output.messages

llm4s builds the OTLP exporter itself, so the OpenTelemetry SDK’s own OTEL_EXPORTER_OTLP_HEADERS variable is not read. Set headers in application.conf, binding secrets as usual:

1
2
3

llm4s.tracing.opentelemetry.headers {
  Authorization = ${?OTEL_AUTH_HEADER}   # e.g. "Bearer <token>"
}

Example Configuration:

1
2
3
4
5
6
7
8
9
10

# Local Jaeger (via Docker)
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-agent
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# Cloud Provider with Authentication
TRACING_MODE=opentelemetry
OTEL_SERVICE_NAME=llm4s-runner
OTEL_EXPORTER_OTLP_ENDPOINT=https://otel-collector.example.com:4317
OTEL_AUTH_HEADER="Bearer sk-otel-token"   # bound by the headers block above

Spans Generated:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

Span: LLM Completion
  ├─ gen_ai.request.model: gpt-4o
  ├─ gen_ai.usage.input_tokens: 150
  └─ gen_ai.usage.output_tokens: 50

Span: Tool Execution: web_search
  ├─ tool.name: web_search
  ├─ tool.input: "Scala 3 features"
  └─ duration_ms: 1234

Span: Token Usage - completion
  ├─ operation: completion
  ├─ gen_ai.usage.total_tokens: 200
  └─ cost.usd: 0.0042

Span: Agent State Updated
  ├─ status: RUNNING
  ├─ message_count: 5
  └─ log_count: 12

Setting Up Jaeger (Local Development):

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

# Start Jaeger all-in-one
docker run -d \
  -p 6831:6831/udp \
  -p 6832:6832/udp \
  -p 5778:5778 \
  -p 16686:16686 \
  -p 14268:14268 \
  -p 14250:14250 \
  -p 9411:9411 \
  jaegertracing/all-in-one:latest

# Configure LLM4S
export TRACING_MODE=opentelemetry
export OTEL_SERVICE_NAME=llm4s-dev
export OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317

# View traces at http://localhost:16686

Code Usage:

1
2
3
4
5
6
7
8
9
10
11
12
13

import org.llm4s.config.Llm4sConfig
import org.llm4s.trace.{ TraceEvent, Tracing }

// Build the tracer that llm4s.tracing.mode selects. fromSettings fails if the mode's
// backend - here llm4s-observability-otel - is not on the classpath; Tracing.create
// would log that and fall back to NoOpTracing instead.
val result = for {
  settings <- Llm4sConfig.tracing()
  tracer   <- Tracing.fromSettings(settings)
  _        <- tracer.traceEvent(TraceEvent.AgentInitialized("query", Vector("tool1")))
  _        <- tracer.traceTokenUsage(usage, "gpt-4o", "completion")
  _         = tracer.shutdown() // always shut down to flush pending spans
} yield ()

Other tracing backends

console and none are built into llm4s-core; every other mode is served by a org.llm4s.trace.spi.TracingBackend that a module registers in META-INF/services/org.llm4s.trace.spi.TracingBackend - langfuse by llm4s-observability, opentelemetry by llm4s-observability-otel. A mode llm4s has no name for, such as TRACING_MODE=datadog, is read as TracingMode.Named("datadog") and goes to whichever backend declares that mode. With no such backend, Tracing.create logs an error listing the available modes and traces nothing. For langfuse and opentelemetry the error also names the module to add, since llm4s knows which of its own modules serves them:

1
2

Tracing mode 'opentelemetry' is configured but no TracingBackend for it is on the classpath.
Add the 'org.llm4s' %% 'llm4s-observability-otel' dependency. Available modes: console, noop.

That hint is only a message: dispatch never consults it, and a third-party mode needs no entry in it. See the observability guide to write one.

Disable Tracing 

1

TRACING_MODE=none

Environment variables llm4s reads

These are the variables bound by a ${?VAR} in some llm4s module’s reference.conf, so they work without any configuration of your own. Each sets the config key beside it, which you can also set in application.conf or with -D.

Variable Config key Bound by
TRACING_MODE llm4s.tracing.mode (console - the default - or none, both built in; langfuse, opentelemetry, or the mode of any other TracingBackend on the classpath) llm4s-core
LANGFUSE_URL, LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY, LANGFUSE_ENV, LANGFUSE_RELEASE, LANGFUSE_VERSION llm4s.tracing.langfuse.* llm4s-observability
OTEL_SERVICE_NAME, OTEL_EXPORTER_OTLP_ENDPOINT llm4s.tracing.opentelemetry.serviceName, .endpoint llm4s-observability-otel
OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT llm4s.tracing.opentelemetry.captureContent (false by default: spans carry no prompt or completion content) llm4s-observability-otel
EMBEDDING_MODEL (or legacy EMBEDDING_PROVIDER) llm4s.embeddings.model (.provider) llm4s-core
LLM4S_TRACING_CONSOLE_CAPTURE_CONTENT llm4s.tracing.console.captureContent (false by default: console tracing prints no prompt or completion content) llm4s-core
LLM4S_EXCHANGE_LOGGING_ENABLED, LLM4S_EXCHANGE_LOGGING_DIR llm4s.exchangeLogging.* llm4s-core
HTTPS_PROXY, NO_PROXY, LLM4S_PROXY_USER, LLM4S_PROXY_PASSWORD llm4s.http.proxy.url, .nonProxyHosts, .username, .password (Corporate networks) llm4s-core
LLM4S_TRUSTSTORE, LLM4S_TRUSTSTORE_PASSWORD, LLM4S_KEYSTORE, LLM4S_KEYSTORE_PASSWORD llm4s.http.tls.trustStore, .trustStorePassword, .keyStore, .keyStorePassword llm4s-core
LLM4S_MODEL_REGISTRY_RESOURCE, LLM4S_MODEL_REGISTRY_FILE, LLM4S_MODEL_REGISTRY_URL llm4s.modelRegistry.* llm4s-core
WORKSPACE_DIR, WORKSPACE_IMAGE, WORKSPACE_PORT llm4s.workspace.* llm4s-core
BRAVE_SEARCH_API_KEY, EXA_API_KEY and the other BRAVE_*, EXA_* variables, DUCK_DUCK_GO_SEARCH_API_URL llm4s.tools.* llm4s-core
OPENAI_API_KEY llm4s.credentials.openai.apiKey (OpenAI chat sections and embeddings) llm4s-openai
AZURE_OPENAI_API_KEY llm4s.credentials.azure.apiKey llm4s-openai
REQUESTY_API_KEY llm4s.credentials.requesty.apiKey llm4s-openai
ANTHROPIC_API_KEY llm4s.credentials.anthropic.apiKey llm4s-anthropic
GOOGLE_API_KEY, else GEMINI_API_KEY llm4s.credentials.gemini.apiKey llm4s-gemini
DEEPSEEK_API_KEY, ZAI_API_KEY, OPENROUTER_API_KEY, MISTRAL_API_KEY, XAI_API_KEY llm4s.credentials.<provider>.apiKey llm4s-openai-compatible
COHERE_API_KEY llm4s.credentials.cohere.apiKey (Cohere chat, the Cohere reranker and Cohere embeddings) llm4s-openai-compatible, llm4s-rag, llm4s-cohere
VOYAGE_API_KEY llm4s.credentials.voyage.apiKey llm4s-voyage
JINA_API_KEY llm4s.credentials.jina.apiKey llm4s-jina
PERPLEXITY_API_KEY llm4s.credentials.perplexity.apiKey llm4s-perplexity
TYPESAFE_API_KEY llm4s.credentials.typesafe.apiKey (the Jev decision client) llm4s-jev
TYPESAFE_BASE_URL, TYPESAFE_DEFAULT_MODEL llm4s.jev.baseUrl, llm4s.jev.model llm4s-jev
OPENAI_EMBEDDING_BASE_URL, OPENAI_EMBEDDING_MODEL llm4s.embeddings.openai.* llm4s-openai
VOYAGE_EMBEDDING_BASE_URL, VOYAGE_EMBEDDING_MODEL llm4s.embeddings.voyage.* llm4s-voyage
JINA_EMBEDDING_BASE_URL, JINA_EMBEDDING_MODEL llm4s.embeddings.jina.* llm4s-jina
COHERE_EMBEDDING_BASE_URL, COHERE_EMBEDDING_MODEL llm4s.embeddings.cohere.* llm4s-cohere
OLLAMA_EMBEDDING_BASE_URL, OLLAMA_EMBEDDING_MODEL llm4s.embeddings.ollama.* llm4s-ollama
RERANK_PROVIDER, COHERE_RERANK_BASE_URL, COHERE_RERANK_MODEL llm4s.rerank.*, read by RerankerConfigLoader llm4s-rag
LLM4S_CHECKPOINTER llm4s.agent.checkpointer.kind (memory - the default, built in - sqlite, postgres, or the kind of any other CheckpointerBackend on the classpath) llm4s-agent
LLM4S_RUNTIME_LABEL llm4s.agent.runtime.claims.label (names the runtime holding a thread’s claim in the logs) llm4s-agent
LLM4S_WORKER_LABEL llm4s.agent.worker.label (names an AgentWorker in its queue leases and logs) llm4s-agent
LLM4S_CHECKPOINTER_SQLITE_PATH llm4s.agent.checkpointer.sqlite.path llm4s-agent-checkpoint-sqlite
LLM4S_CHECKPOINTER_POSTGRES_URL, LLM4S_CHECKPOINTER_POSTGRES_USER, LLM4S_CHECKPOINTER_POSTGRES_PASSWORD llm4s.agent.checkpointer.postgres.url, .user, .password llm4s-agent-checkpoint-postgres
PGVECTOR_HOST, PGVECTOR_PORT, PGVECTOR_DATABASE, PGVECTOR_USER, PGVECTOR_PASSWORD, PGVECTOR_TABLE, … llm4s.rag.permissions.pg.* llm4s-rag
ELASTICSEARCH_URL, ELASTICSEARCH_INDEX, ELASTICSEARCH_API_KEY, ELASTICSEARCH_USERNAME, ELASTICSEARCH_PASSWORD llm4s.vectorstore.elasticsearch.*, read by ElasticsearchVectorStoreConfigLoader llm4s-vectorstore-elasticsearch
OPENSEARCH_URL, OPENSEARCH_INDEX, OPENSEARCH_USERNAME, OPENSEARCH_PASSWORD, OPENSEARCH_TOKEN llm4s.vectorstore.opensearch.*, read by OpenSearchVectorStoreConfigLoader llm4s-vectorstore-elasticsearch
REDIS_HOST, REDIS_PORT, REDIS_TLS, REDIS_USERNAME, REDIS_PASSWORD, REDIS_VECTOR_INDEX llm4s.vectorstore.redis.*, read by RedisVectorStoreConfigLoader llm4s-vectorstore-redis

Not read by llm4s unless your application.conf binds them: LLM_MODEL, LLM4S_PROVIDER, every chat provider’s endpoint and model variable (AZURE_API_BASE, AZURE_API_VERSION, OPENAI_BASE_URL, OLLAMA_BASE_URL, …), AZURE_API_KEY (the Azure key is AZURE_OPENAI_API_KEY, the openai SDK’s name), CO_API_KEY, anything for the generic openai-compatible provider, and OTEL_EXPORTER_OTLP_HEADERS. A shared key sets only apiKey: baseUrl, endpoint, model and apiVersion always come from the section. LLM_MODEL was removed with legacy single-provider loading in #903.

The embeddings sample’s variables - EMBEDDING_INPUT_PATH, EMBEDDING_INPUT_PATHS, EMBEDDING_QUERY, CHUNK_SIZE, CHUNK_OVERLAP, CHUNKING_ENABLED, MAX_ROWS_PER_FILE, TOP_DIMS_PER_ROW, GLOBAL_TOPK, SHOW_GLOBAL_TOP, COLOR, TABLE_WIDTH, ENABLE_EXPERIMENTAL_STUBS and LLM4S_EMBEDDINGS_LOCAL_{IMAGE,AUDIO,VIDEO}_MODEL - are bound by the samples’ own application.conf under llm4s.samples.embeddings, not by a library module (#2265).

The example agent service’s variables - PORT, LLM4S_SERVICE_MODEL, LLM4S_SERVICE_TOOL_DELAY and LLM4S_SERVICE_METRICS - are bound by modules/deploy-service’s own reference.conf (unpublished) under llm4s.deploy-service; see deploy/README.md.

Environment-specific configuration

Keep the provider structure in HOCON and inject secrets from the environment. One application.conf can serve every environment, since only the selected section’s variables need to be set, or you can keep one file per environment:

1
2
3
4
5
6
7
8
9
10
11
12

# src/main/resources/application.conf - development
llm4s {
  providers {
    provider = "ollama-local"
    ollama-local {
      provider = "ollama"
      model    = "llama3.2"
      baseUrl  = "http://localhost:11434"
    }
  }
  tracing.mode = "console"
}

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

# src/main/resources/prod.conf - production
include "application.conf"

llm4s {
  providers = null                     # drop the development sections
  providers {
    provider = "claude"
    claude {
      provider = "anthropic"
      model    = "claude-sonnet-4-20250514"
      apiKey   = ${?ANTHROPIC_API_KEY}   # explicit in production: which account this bills
    }
  }
  tracing.mode = "langfuse"            # llm4s-observability; LANGFUSE_PUBLIC_KEY / LANGFUSE_SECRET_KEY from the environment
}

1

java -Dconfig.resource=prod.conf -jar app.jar

providers = null matters: without it the included ollama-local section would stay, and be validated, in production.

Configuration Best Practices

✅ DO

  1. Keep secrets in environment variables - the vendor’s own variable (OPENAI_API_KEY), or one you bind with apiKey = ${?VAR} in a section; in production, give each section its own
  2. Keep structure in application.conf: provider sections, models, defaults
  3. Add .env files to .gitignore if you use them to export variables
  4. Use different configs for dev/staging/prod
  5. Validate configuration at startup
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17

import org.llm4s.config.Llm4sConfig

object ApplicationBoundary {
  def validateAndStart(): Unit = {
    // Fail fast at startup if config is invalid
    Llm4sConfig.defaultProvider().fold(
      error => {
        Console.err.println(s"FATAL: Configuration error: ${error.formatted}")
        System.exit(1)
      },
      config => {
        println(s"Configuration loaded for model: ${config.model}")
        // startApplication(config)
      }
    )
  }
}

❌ DON’T

  1. Don’t hardcode API keys in source code or application.conf
  2. Don’t commit .env files to version control
  3. Don’t use System.getenv() directly (use Llm4sConfig)
  4. Don’t mix production keys in development
  5. Don’t skip validation - fail fast on bad config

Validation Example 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMConnect
import org.llm4s.model.ModelRegistryService

object ValidateConfig extends App {
  println("Validating LLM4S configuration...")

  // Perform validation at the application edge
  val validationResult = for {
    providerConfig <- Llm4sConfig.defaultProvider()
    registry       <- Llm4sConfig.modelRegistryService()
    given ModelRegistryService = registry
    client <- LLMConnect.getClient(providerConfig)
  } yield (providerConfig, client)

  validationResult match {
    case Right((config, client)) =>
      println(s"✅ Model: ${config.model}")
      println(s"✅ Provider: ${config.getClass.getSimpleName}")
      println(s"✅ Client ready: ${client.getClass.getSimpleName}")
      println("✅ Configuration valid!")

    case Left(error) =>
      Console.err.println(s"❌ Configuration error: ${error.formatted}")
      System.exit(1)
  }
}

Troubleshooting

Problem: “missing required fields: apiKey”

Symptoms:

1
2

ConfigurationError: Provider 'openai-main' (provider = openai) is missing required fields:
  - apiKey: set OPENAI_API_KEY, or set apiKey under llm4s.providers.openai-main in application.conf

Root causes:

  1. Neither the vendor’s variable (OPENAI_API_KEY) nor a section apiKey is set in this process
  2. The provider’s module is missing, so nothing binds its variable (see the table in Named provider sections)
  3. The section binds its own variable (apiKey = ${?OTHER_VAR}) that is unset, and the vendor’s variable is unset too
  4. .env file not loaded in the shell that starts the JVM

The error names the section it is about, and only a section being loaded is validated. If it names a section you did not ask for, check llm4s.providers.provider (and any -Dllm4s.providers.provider override): that is the section defaultProvider() loads. The one call that validates every section is Llm4sConfig.providers().

Debug steps:

1
2
3
4
5

# Is the vendor's variable set in this shell?
echo $OPENAI_API_KEY

# Does the section set a key of its own? (an apiKey line wins over OPENAI_API_KEY)
grep -n apiKey src/main/resources/application.conf

Problem: “Configured provider ‘…’ was not found”

Symptoms:

1

ConfigurationError: Configured provider 'openai-main' was not found

Fix: llm4s.providers.provider (or the name passed to Llm4sConfig.provider(...)) must match a section name exactly. Check for a typo, and for an LLM4S_PROVIDER-style override selecting a section this file does not define. If llm4s.providers.provider is absent altogether, defaultProvider() fails with “No default provider configured under llm4s.providers.provider” - set it, or call Llm4sConfig.provider("<name>").

Problem: “Unknown provider” / provider not registered

Symptoms: an error naming llm4s.providers.<name>.provider and listing the registered providers.

Fix: the section’s provider = "..." id comes from a module that is not on the classpath. Add it (see the table in Named provider sections), or fix the id’s spelling.

Problem: “Configuration not loading from application.conf”

Symptoms: defaultProvider() behaves as if your sections were not there.

Debug:

1
2
3
4
5

import com.typesafe.config.ConfigFactory

val config = ConfigFactory.load()
println(config.hasPath("llm4s.providers.provider"))    // Should be true
println(config.getString("llm4s.providers.provider"))  // Should name your section

Common causes:

  • application.conf not in src/main/resources/ (or not on the runtime classpath)
  • -Dconfig.file / -Dconfig.resource pointing somewhere else
  • HOCON syntax error (missing quotes, wrong nesting)
  • A -D system property overriding the key (system properties have the highest precedence)

Problem: “Provider mismatch”

Symptoms:

1
2
3

import org.llm4s.error.AuthenticationError

Left(AuthenticationError("Invalid API key"))

Root cause: the section’s own apiKey binds a variable holding another provider’s key, or the vendor’s variable holds the wrong key. The INFO line `llm4s.providers.: API key from

` says which place the key came from. ```hocon # ❌ Wrong: an OpenAI section reading the Anthropic key openai-main { provider = "openai" model = "gpt-4o" apiKey = ${?ANTHROPIC_API_KEY} } ``` ### Problem: "Rate limiting / 429 errors" **Symptoms:** ``` import org.llm4s.error.RateLimitError Left(RateLimitError("Rate limit exceeded")) ``` **Solutions:** 1. **Add retry logic:** ```scala import scala.concurrent.duration._ import org.llm4s.error.RateLimitError import org.llm4s.types.Result // Note: This example uses Thread.sleep for simplicity. // In production, prefer scala.concurrent or cats-effect for non-blocking delays. def retryWithBackoff[A](op: => Result[A], maxRetries: Int = 3): Result[A] = { (1 to maxRetries).foldLeft(op) { (result, attempt) => result match { case Left(_: RateLimitError) if attempt < maxRetries => Thread.sleep(Math.pow(2, attempt).toLong * 1000) // Exponential backoff op case other => other } } } ``` 2. **Use a cheaper model for testing** - change the section's `model`: ```hocon openai-main { provider = "openai" model = "gpt-4o-mini" # development; gpt-4o in production } ``` 3. **Switch to Ollama for development** - point `llm4s.providers.provider` at an `ollama-local` section (see [Switching providers](#switching-providers)); free, no rate limits. ### Problem: "Slow responses or timeouts" **Symptoms:** Requests take 30+ seconds or timeout. **Solutions:** 1. **Raise (or lower) the timeouts.** Each client has a default (see [Timeouts](#timeouts)): two minutes for a completion, and five or ten minutes for a stream. A section's `timeouts { request = 5m, stream = 20m }` block changes them per provider. 2. **Use streaming for long responses:** ```scala // Non-streaming: waits for full response client.complete(Conversation(Seq(UserMessage("Your query")))) // Streaming: get tokens as they arrive client.streamComplete( Conversation(Seq(UserMessage("Your query"))), onChunk = chunk => chunk.content.foreach(print) ) ``` 3. **Reduce response length:** ```scala client.complete( Conversation(Seq(UserMessage("Your query"))), CompletionOptions(maxTokens = Some(500)) // Limit response length ) ``` ### Problem: "OutOfMemoryError with embeddings" **Symptoms:** JVM crashes when processing large document collections. **Solutions:** 1. **Batch embeddings instead of loading all at once:** ```scala import org.llm4s.llmconnect.model.EmbeddingRequest val documents: List[String] = loadDocuments() val batchSize = 100 val embeddings = documents.grouped(batchSize).flatMap { batch => embedder.embed(EmbeddingRequest(batch, embeddingModel)) match { case Right(response) => response.embeddings case Left(err) => println(s"Batch failed: $err") List.empty } }.toList ``` 2. **Increase JVM heap:** ```bash export SBT_OPTS="-Xmx4G -Xss4M" sbt run ``` 3. **Use a local embedding model (smaller memory footprint):** ```bash EMBEDDING_MODEL=ollama/nomic-embed-text ``` --- ## Next Steps Configuration complete! Now you can: 1. **[Explore features →](next-steps)** - Dive into agents, tools, and more 2. **[Browse examples →](/examples/)** - See configuration in action 3. **[User guide →](../guide/basic-usage)** - Learn core concepts 4. **[Observability →](../guide/observability)** - Set up tracing --- ## Additional Resources - **[Llm4sConfig API](/api/#4-explicit-configuration)** - Detailed API documentation - **[Environment variables llm4s reads](#environment-variables-llm4s-reads)** - Complete list - **[Production Guide](../PRODUCTION_DEPLOYMENT)** - Production best practices - **[Discord Community](https://discord.gg/4uvTPn6qww)** - Get help with configuration --- **Ready to build?** [Explore what's next →](next-steps)