org.llm4s.testkit

Proves a provider module's reference.conf binds what its descriptor declares.

A descriptor's configSpec.apiKeyEnv names the variables its module binds to llm4s.credentials.<id>.apiKey, and the binding itself is a reference.conf line: nothing but a test keeps the two in step. These functions load a section or embeddings block that sets no apiKey of its own - so the shared credential is the only place a key can come from - against every reference.conf on the test classpath and an injected environment (the real one switched off).

ProviderModuleChecks.assertCredentialBindings wraps chatBindings as an assertion; use these directly for the cases it does not cover - an alias, two variables in precedence order, a provider that must ''not'' pick up a neighbour's variable.

A local HTTP server to point a provider client at, so a test exercises the client's real request and response handling - including streaming - without a network or an API key.

It is the JDK's com.sun.net.httpserver.HttpServer on an ephemeral port, so it adds no dependency.

withServer("/v1/chat")(exchange => sendSseResponse(exchange, body)) { baseUrl =>
 val client = assertBuildsClient(AcmeProvider, section.copy(baseUrl = Some(BaseUrl(baseUrl))))
 assertStreams(client)
}

The checks every provider module's Llm4s<Name>ModuleSpec makes, as assertions.

A provider module is a dependency, not an edit to llm4s: its Llm4sProviderModule is found through META-INF/services, and a mistake there - a typo in the services file, an object where a class is needed, a second module claiming the same id, a reference.conf binding the wrong variable - fails at a user's runtime, not at your compile time. These checks catch each of those in your own build. They are the ones every provider module in the llm4s repository runs.

Mix the trait into a spec of any ScalaTest style, or call the companion object's methods. A failed check fails the test at the call site with a message saying what was wrong:

class Llm4sAcmeModuleSpec extends AnyWordSpec with Matchers with ProviderModuleChecks:

 "llm4s-acme" should {
   "register itself through META-INF/services" in assertModule(new Llm4sAcmeModule)

   "build a client from a section" in {
     assertBuildsClient(AcmeProvider, AcmeProvider.section(...))
     assertRefusesForeignConfig(AcmeProvider)
   }

   "bind ACME_API_KEY" in assertCredentialBindings(AcmeProvider)
 }

Modules are compared by class and descriptors by equality (normally reference equality on an object), so pass the same descriptor instances your module lists.

The checks, for a spec that would rather call them than mix them in.

Loads provider configuration the way an application does, with the environment supplied by the test.

Each method reads hocon - what the application would write in its application.conf - over every reference.conf on the test classpath, your module's included, and resolves ${?VAR} substitutions against env alone. The real process environment is switched off, so a developer machine with ACME_API_KEY exported cannot make a test pass that fails in CI, and a test can prove your reference.conf binds a variable without setting it.

given ProviderRegistry = ProviderRegistry.default

ProviderTestConfig.loadProvider(
 "acme-main",
 """llm4s.providers.acme-main { provider = "acme", model = "acme-large" }""",
 Map("ACME_API_KEY" -> "test-key")
) // Right(AcmeConfig(...)), the key taken from llm4s.credentials.acme.apiKey

The provider a section names is resolved through the ProviderRegistry in scope: ProviderRegistry.default for what discovery finds on the classpath, or ProviderRegistry.ofModules(new MyModule) to pin it.