RAG in Scala

To answer questions over your own documents in Scala, use llm4s-rag: one RAG pipeline chunks and embeds what you ingest, stores the vectors, retrieves the passages that match a question and has a model answer from them, with the passages it used. The vectors live in memory, in SQLite or in PostgreSQL with pgvector, and the quality can be measured with RAGAS metrics.

  1. Add the module
  2. Build a pipeline
  3. Ingest and ask
  4. Other stores
  5. Evaluate
  6. Going further

Add the module 

1
2
3
4

libraryDependencies ++= Seq(
  "org.llm4s" %% "llm4s-rag"    % "0.5.1",
  "org.llm4s" %% "llm4s-openai" % "0.5.1"
)

The provider module supplies the embedding model as well as the chat model: OpenAI here; Ollama, Voyage AI, Jina AI and Cohere embeddings have modules of their own (embeddings). The embedding model is chosen in configuration, for example EMBEDDING_MODEL=openai/text-embedding-3-small.

Build a pipeline 

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import org.llm4s.config.Llm4sConfig
import org.llm4s.llmconnect.LLMClient
import org.llm4s.model.ModelRegistryService
import org.llm4s.rag.RAG
import org.llm4s.rag.RAG.RAGConfigOps
import org.llm4s.types.Result

def docsPipeline(client: LLMClient): Result[RAG] = for {
  registryService <- Llm4sConfig.modelRegistryService()
  given ModelRegistryService = registryService
  selected <- Llm4sConfig.embeddings()
  (provider, embeddingConfig) = selected
  rag <- RAG
    .builder()
    .withEmbeddings(provider)
    .withPgVector()
    .withLLM(client)
    .build(_ => Right(embeddingConfig))
} yield rag

withPgVector() keeps the vectors in PostgreSQL with the pgvector extension, on localhost:5432 (its overloads take a connection string, user, password and table). Leave it out and the pipeline keeps everything in memory, which suits a test or a demo; withSQLite(path) keeps it in a file. client is the chat client that writes the answers, built as on the first-call page.

Ingest and ask 

1
2
3
4
5
6
7
8

import org.llm4s.rag.{ RAG, RAGAnswerResult }
import org.llm4s.types.Result

def askTheDocs(rag: RAG, question: String): Result[RAGAnswerResult] = for {
  _      <- rag.ingestText("Opaque types hide their representation outside the defining scope.", "opaque-types")
  _      <- rag.ingestText("Given instances supply the values of context parameters.", "givens")
  answer <- rag.queryWithAnswer(question)
} yield answer

queryWithAnswer returns the answer and the passages it was given (contexts), so you can show sources. Instead of text, rag.ingest(path) reads a file or a whole directory, and rag.query(question) returns the passages without asking the model. Retrieval is hybrid: vector similarity and keyword (BM25) search, fused, with optional reranking by Cohere or by a model (vector store guide).

Other stores

Besides the pipeline’s own storage, the VectorStore API has backends for Qdrant, and, in separate modules, Elasticsearch, OpenSearch, Redis and Valkey. They are covered in the vector store guide.

Evaluate

A RAG pipeline needs measuring, not eyeballing. LLM4S computes the RAGAS metrics (faithfulness, answer relevancy, context precision and context recall) and has a benchmark harness that compares chunking strategies, fusion strategies and embedding providers on a dataset: see RAG evaluation.

Going further