LLMClientIO

org.llm4s.effect.cats.LLMClientIO
See theLLMClientIO companion object
trait LLMClientIO[F[_]]

cats-effect wrapper for LLMClient.

All blocking LLM calls are shifted to the blocking thread pool via Async[F].interruptible, keeping the compute pool free for fibers. Cancelling the fiber (or a timeout) interrupts the provider thread, which is how llm4s providers are cancelled: they keep the interrupt and return Left(CancelledError). LLMError values are surfaced as LLMException in the F error channel.

Attributes

Companion
object
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Abstract methods

def agent(): AgentIO[F]

Creates an AgentIO backed by this client.

Creates an AgentIO backed by this client.

Attributes

def complete(conversation: Conversation, options: CompletionOptions): F[Completion]
def streamComplete(conversation: Conversation, options: CompletionOptions): Stream[F, StreamedChunk]

Streams chunks incrementally as an fs2 Stream.

Streams chunks incrementally as an fs2 Stream.

The underlying blocking call runs on an interruptible blocking thread and pushes each chunk into a bounded queue (backpressure: the provider thread blocks while the consumer lags). Chunks are delivered as they arrive, not after the call completes. If the call fails mid-stream, chunks already received are emitted first and the stream then fails with LLMException. Stopping consumption early (e.g. take) or cancelling the fiber interrupts the blocking call.

Attributes