llm4s-docs
API
Generated with
llm4s-docs/org.llm4s/org.llm4s.ox/LLMClientOx

LLMClientOx

org.llm4s.ox.LLMClientOx
@Experimental
object LLMClientOx

Direct-style operations over an LLMClient, built on SoftwareMill's Ox: a streamed completion as an Ox Flow, bounded fan-out and a race between providers. import org.llm4s.ox.* brings them into scope.

Core's calls already block the calling thread and return a Result; interrupting that thread cancels the call, which then returns Left(CancelledError). These operations keep that model and add what plain direct style leaves to the caller: a pull-based stream, a parallelism bound and the interruption of work nobody waits for. Every fork runs on a JDK 21 virtual thread.

Attributes

Graph
Supertypes
class Object
trait Matchable
class Any
Self type

Members list

Value members

Concrete methods

def raceProviders(clients: LLMClient*)(conversation: Conversation, options: CompletionOptions): Result[Completion]

Completes conversation with every client at once and returns the first success; the calls still running are interrupted (core returns CancelledError from each) and the race returns once they have ended. When every call fails, the result is the first failure to arrive. With no client, a ValidationError.

Completes conversation with every client at once and returns the first success; the calls still running are interrupted (core returns CancelledError from each) and the race returns once they have ended. When every call fails, the result is the first failure to arrive. With no client, a ValidationError.

Attributes

Extensions

Extensions

extension (client: LLMClient)
def parComplete(conversations: Seq[Conversation], parallelism: Int, options: CompletionOptions): Vector[Result[Completion]]

Completes each conversation with client, at most parallelism at a time (a bound below 1 counts as 1), on virtual threads. The results are in the order of conversations, one Result each: a failed request does not cancel the others. For all-or-nothing fan-out, use Ox's parEitherLimit with client.complete.

Completes each conversation with client, at most parallelism at a time (a bound below 1 counts as 1), on virtual threads. The results are in the order of conversations, one Result each: a failed request does not cancel the others. For all-or-nothing fan-out, use Ox's parEitherLimit with client.complete.

Attributes

def streamFlow(conversation: Conversation, options: CompletionOptions)(using BufferCapacity): Flow[StreamedChunk]

The completion of conversation, streamed as an Ox Flow of the provider's chunks, in order.

The completion of conversation, streamed as an Ox Flow of the provider's chunks, in order.

The flow is cold: each run makes one streamComplete call. The call runs on a virtual thread of its own and hands each chunk to the consumer through a channel of the given BufferCapacity (Ox's default is 16): when the consumer lags, the provider's thread waits. When the consumer stops early (a take, a failure, the end of its scope), the run's scope interrupts the provider call, which core turns into CancelledError, and waits for it to return, so no thread outlives the run.

A provider Left fails the flow with an LLMException carrying the LLMError, after the chunks delivered before it. accumulate turns it back into a Left.

Attributes

extension (flow: Flow[StreamedChunk])
def accumulate: Result[Completion]

Runs the flow and accumulates its chunks into the Completion they describe (content, tool calls, thinking, finish), as core's StreamingAccumulator does. A chunk carries no token usage or model, so the result has none. A flow failed by an LLMException returns its LLMError as a Left; any other exception, an interrupt included, propagates.

Runs the flow and accumulates its chunks into the Completion they describe (content, tool calls, thinking, finish), as core's StreamingAccumulator does. A chunk carries no token usage or model, so the result has none. A flow failed by an LLMException returns its LLMError as a Left; any other exception, an interrupt included, propagates.

Attributes

def textDeltas: Flow[String]

The text of each chunk that carries some, in order; tool-call, thinking and finish chunks are dropped.

The text of each chunk that carries some, in order; tool-call, thinking and finish chunks are dropped.

Attributes

Generated with