RagStreams

org.llm4s.javaapi.reactor.RagStreams
object RagStreams

A RAG answer, with its sources, as a Reactor Flux of RagAnswerEvents (#2727): for a Spring WebFlux endpoint that sends the citations over SSE first, then the answer as it is generated. It needs llm4s-java-rag on the classpath, which an application holding a JRag has.

Each method runs JRag.askStream once per subscription, so the Flux:

  • is cold: each subscription makes its own search and model call, on a thread of its own;
  • emits one RagAnswerEvent.Sources, then a RagAnswerEvent.Chunk for each piece of the answer's text, then one RagAnswerEvent.Complete with the whole answer, then completes;
  • ends with onError for a failure - of the search, the model, or a pipeline that refuses the plain query - carrying an LlmException whose getKind() says what failed, after the elements already delivered;
  • honours backpressure: a subscriber that requests less than the model produces holds the call's thread rather than losing text;
  • cancelling the subscription (dispose(), take(n), a client that disconnects) interrupts the call, as JRagAnswerStream.cancel() does, without waiting for it.

The sources and the answer travel as elements of one Flux, not as a Mono beside it, so they cannot be subscribed to apart - two subscriptions would make two searches - and their order is the stream's own.

A null argument, a blank question or a pipeline without a chat client is never thrown: the Flux fails on subscription with an LlmException of kind VALIDATION or CONFIGURATION.

Flux<RagAnswerEvent> events = RagStreams.toFlux(rag, "How long do refunds take?");
events.subscribe(event -> {
   switch (event) {
       case RagAnswerEvent.Sources s -> showSources(s.sources());
       case RagAnswerEvent.Chunk c -> System.out.print(c.text());
       case RagAnswerEvent.Complete c -> System.out.println();
   }
});

Attributes

Source
RagStreams.scala
Graph
Supertypes
class Object
trait Matchable
class Any
Self type
RagStreams.type

Members list

Value members

Concrete methods

def toFlux(rag: JRag, question: String): Flux[RagAnswerEvent]

The answer to question, from at most the pipeline's topK chunks, as a Flux of events.

The answer to question, from at most the pipeline's topK chunks, as a Flux of events.

Attributes

Source
RagStreams.scala
def toFlux(rag: JRag, question: String, topK: Int): Flux[RagAnswerEvent]

The answer to question, from at most topK chunks, at least 1, as a Flux of events.

The answer to question, from at most topK chunks, at least 1, as a Flux of events.

Attributes

Source
RagStreams.scala