llm4s-core
API
Generated with
llm4s-core/org.llm4s/org.llm4s.agent/org.llm4s.agent.guardrails/org.llm4s.agent.guardrails.rag

org.llm4s.agent.guardrails.rag

package org.llm4s.agent.guardrails.rag

Members list

Type members

Classlikes

class ContextRelevanceGuardrail(val llmClient: LLMClient, val threshold: Double, val minRelevantRatio: Double, val onFail: GuardrailAction) extends RAGGuardrail

LLM-based guardrail to validate that retrieved chunks are relevant to the query.

LLM-based guardrail to validate that retrieved chunks are relevant to the query.

ContextRelevanceGuardrail uses an LLM to evaluate whether the chunks retrieved from a vector store are actually relevant to the user's original query. This is critical for RAG quality - retrieving irrelevant chunks leads to poor answers.

Evaluation process:

  1. Each chunk is evaluated for relevance to the query
  2. Relevance is scored from 0.0 (completely irrelevant) to 1.0 (highly relevant)
  3. Overall score is computed as average chunk relevance
  4. Context passes if enough chunks are relevant

Use cases:

  • Detect retrieval failures before generating responses
  • Filter out irrelevant chunks before sending to LLM
  • Measure and monitor retrieval quality

Example usage:

val guardrail = ContextRelevanceGuardrail(llmClient, threshold = 0.6)

val context = RAGContext(
 query = "What are the symptoms of diabetes?",
 retrievedChunks = Seq(
   "Diabetes symptoms include increased thirst...",
   "The history of the Roman Empire..." // Irrelevant
 )
)

// Validate that retrieved context is relevant
guardrail.validateWithContext(response, context)

Value parameters

llmClient

The LLM client for evaluation

minRelevantRatio

Minimum ratio of relevant chunks required (default: 0.5)

onFail

Action to take when relevance is insufficient (default: Block)

threshold

Minimum relevance score for a chunk to be considered relevant (default: 0.5)

Attributes

Companion
object
Supertypes
trait RAGGuardrail
trait Guardrail[String]
class Object
trait Matchable
class Any
Show all
object ContextRelevanceGuardrail

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
final case class ContextRelevanceResult(overallScore: Double, chunkScores: Seq[Double], relevantChunkCount: Int, explanation: String)

Result of context relevance evaluation.

Result of context relevance evaluation.

Value parameters

chunkScores

Individual relevance scores for each chunk

explanation

Brief explanation of the evaluation

overallScore

Overall relevance score (0.0 to 1.0)

relevantChunkCount

Number of chunks meeting the threshold

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
class GroundingGuardrail(val llmClient: LLMClient, val threshold: Double, val onFail: GuardrailAction, val strictMode: Boolean) extends RAGGuardrail

LLM-based grounding guardrail for RAG validation.

LLM-based grounding guardrail for RAG validation.

GroundingGuardrail uses an LLM to evaluate whether a response is factually grounded in the retrieved context. This is critical for RAG applications to prevent hallucination and ensure answer quality.

Evaluation process:

  1. Each claim in the response is checked against the retrieved chunks
  2. Claims are classified as: supported, not supported, or contradicted
  3. An overall grounding score is computed
  4. Response passes if score >= threshold

Scoring:

  • 1.0: All claims are fully supported by the context
  • 0.5-0.9: Most claims supported, some unverifiable
  • 0.0-0.4: Many claims not supported or contradicted

Example usage:

val guardrail = GroundingGuardrail(llmClient, threshold = 0.8)

// Use in RAG pipeline
val context = RAGContext(
 query = "What causes climate change?",
 retrievedChunks = Seq(
   "Greenhouse gases trap heat in the atmosphere...",
   "Human activities release CO2 and methane..."
 )
)

guardrail.validateWithContext(response, context)

Value parameters

llmClient

The LLM client for evaluation

onFail

Action to take when grounding fails (default: Block)

strictMode

If true, ANY ungrounded claim fails. If false, uses score threshold.

threshold

Minimum grounding score to pass (default: 0.7)

Attributes

Companion
object
Supertypes
trait RAGGuardrail
trait Guardrail[String]
class Object
trait Matchable
class Any
Show all
object GroundingGuardrail

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
final case class GroundingResult(score: Double, isGrounded: Boolean, ungroundedClaims: Seq[String], explanation: String)

Result of a grounding evaluation.

Result of a grounding evaluation.

Value parameters

explanation

Brief explanation of the evaluation

isGrounded

Whether the response passes the threshold

score

Overall grounding score (0.0 to 1.0)

ungroundedClaims

Claims that weren't supported by context (if any)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
final case class RAGContext(query: String, retrievedChunks: Seq[String], sources: Seq[String])

Context provided to RAG guardrails for enhanced validation.

Context provided to RAG guardrails for enhanced validation.

RAGContext encapsulates all the information a guardrail needs to validate a response in the context of a RAG (Retrieval-Augmented Generation) workflow.

Value parameters

query

The original user query

retrievedChunks

The chunks retrieved from the vector store

sources

Optional source identifiers for each chunk (file paths, URLs, etc.)

Attributes

Companion
object
Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
object RAGContext

Attributes

Companion
class
Supertypes
trait Product
trait Mirror
class Object
trait Matchable
class Any
Self type
RAGContext.type
trait RAGGuardrail extends OutputGuardrail

Base trait for RAG-specific guardrails that need retrieval context.

Base trait for RAG-specific guardrails that need retrieval context.

RAGGuardrail extends OutputGuardrail with additional context-aware validation methods. This allows guardrails to validate responses against the retrieved chunks from a RAG pipeline, enabling:

  • Grounding checks: Verify the response is supported by retrieved context
  • Context relevance: Check if retrieved chunks are relevant to the query
  • Source attribution: Ensure citations are accurate and present

RAG guardrails should implement validateWithContext for full RAG validation. The standard validate method provides a fallback for non-RAG usage.

Example usage:

// Create a grounding guardrail
val grounding = GroundingGuardrail(llmClient, threshold = 0.8)

// Validate with RAG context
val context = RAGContext(
 query = "What is photosynthesis?",
 retrievedChunks = Seq("Plants convert sunlight...", "Chlorophyll absorbs...")
)
grounding.validateWithContext("Photosynthesis converts light to energy", context)

Attributes

See also

GroundingGuardrail for factuality validation

ContextRelevanceGuardrail for chunk relevance checks

SourceAttributionGuardrail for citation verification

Companion
object
Supertypes
trait Guardrail[String]
class Object
trait Matchable
class Any
Known subtypes
object RAGGuardrail

Companion object with utilities for RAG guardrails.

Companion object with utilities for RAG guardrails.

Attributes

Companion
trait
Supertypes
class Object
trait Matchable
class Any
Self type
object RAGGuardrails

Preset configurations for RAG guardrails.

Preset configurations for RAG guardrails.

RAGGuardrails provides convenient preset combinations of guardrails for common RAG (Retrieval-Augmented Generation) use cases. Each preset balances security, quality, and latency differently.

Preset Levels:

  • minimal: Basic safety only (PII, input length)
  • standard: Balanced protection for production use
  • strict: Maximum safety with comprehensive validation
  • monitoring: Full validation in warn mode (no blocking)

Example usage:

// Get standard guardrails for production
val (inputGuardrails, outputGuardrails, ragGuardrails) =
 RAGGuardrails.standard(llmClient)

// Use in agent
agent.run(
 query = userQuery,
 tools = tools,
 inputGuardrails = inputGuardrails,
 outputGuardrails = outputGuardrails
)

// Use RAG guardrails separately
ragGuardrails.foreach { guardrail =>
 guardrail.validateWithContext(response, ragContext)
}

Attributes

Supertypes
class Object
trait Matchable
class Any
Self type
class SourceAttributionGuardrail(val llmClient: LLMClient, val requireAttributions: Boolean, val minAttributionScore: Double, val onFail: GuardrailAction) extends RAGGuardrail

LLM-based guardrail to validate that responses properly cite their sources.

LLM-based guardrail to validate that responses properly cite their sources.

SourceAttributionGuardrail ensures that RAG responses include proper citations to the source documents from which information was derived. This is important for transparency, verifiability, and trust.

Evaluation criteria:

  • Does the response cite sources for factual claims?
  • Are the citations accurate (pointing to the right chunks)?
  • Are all major claims properly attributed?

Use cases:

  • Ensure transparency in RAG responses
  • Enable users to verify information
  • Comply with requirements for attributing sources
  • Detect when responses fail to cite available sources

Example usage:

val guardrail = SourceAttributionGuardrail(llmClient)

val context = RAGContext.withSources(
 query = "What causes climate change?",
 chunks = Seq("Human activities release greenhouse gases..."),
 sources = Seq("IPCC Report 2023.pdf")
)

// Response should cite sources
val response = "According to the IPCC Report, human activities release greenhouse gases..."
guardrail.validateWithContext(response, context)

Value parameters

llmClient

The LLM client for evaluation

minAttributionScore

Minimum attribution quality score (default: 0.5)

onFail

Action to take when attribution is insufficient (default: Block)

requireAttributions

Whether citations are required (default: true)

Attributes

Companion
object
Supertypes
trait RAGGuardrail
trait Guardrail[String]
class Object
trait Matchable
class Any
Show all
object SourceAttributionGuardrail

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
final case class SourceAttributionResult(hasAttributions: Boolean, attributionScore: Double, citedSources: Seq[String], uncitedClaims: Seq[String], explanation: String)

Result of source attribution evaluation.

Result of source attribution evaluation.

Value parameters

attributionScore

Score for quality of attributions (0.0 to 1.0)

citedSources

List of sources that were cited in the response

explanation

Brief explanation of the evaluation

hasAttributions

Whether the response contains any source citations

uncitedClaims

Claims that should have been cited but weren't

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
class TopicBoundaryGuardrail(val llmClient: LLMClient, val allowedTopics: Seq[String], val threshold: Double, val onFail: GuardrailAction) extends InputGuardrail

LLM-based guardrail to ensure queries stay within allowed topic boundaries.

LLM-based guardrail to ensure queries stay within allowed topic boundaries.

TopicBoundaryGuardrail validates that user queries are within the scope of allowed topics for the RAG application. This is useful for:

  • Keeping conversations focused on the intended domain
  • Preventing misuse of specialized assistants
  • Ensuring the knowledge base is appropriate for the query

Evaluation process:

  1. The query is analyzed to determine its topic/intent
  2. The detected topic is compared against allowed topics
  3. Query passes if it matches at least one allowed topic

Use cases:

  • Domain-specific assistants (medical, legal, technical)
  • Customer support bots with defined scope
  • Knowledge bases with specific subject matter
  • Compliance with usage policies

Example usage:

val guardrail = TopicBoundaryGuardrail(
 llmClient,
 allowedTopics = Seq("scala programming", "functional programming", "software development"),
 threshold = 0.6
)

// On-topic query
guardrail.validate("How do I use pattern matching in Scala?") // passes

// Off-topic query
guardrail.validate("What's the best pizza restaurant?") // fails

Value parameters

allowedTopics

List of topics that queries should relate to

llmClient

The LLM client for evaluation

onFail

Action to take when query is off-topic (default: Block)

threshold

Minimum relevance score to be considered on-topic (default: 0.5)

Attributes

Companion
object
Supertypes
trait Guardrail[String]
class Object
trait Matchable
class Any
object TopicBoundaryGuardrail

Attributes

Companion
class
Supertypes
class Object
trait Matchable
class Any
Self type
final case class TopicBoundaryResult(isOnTopic: Boolean, relevanceScore: Double, matchedTopics: Seq[String], detectedTopic: String, explanation: String)

Result of topic boundary evaluation.

Result of topic boundary evaluation.

Value parameters

detectedTopic

The actual topic detected in the query

explanation

Brief explanation of the evaluation

isOnTopic

Whether the query is within allowed topics

matchedTopics

Topics from the allowed list that match the query

relevanceScore

Score for topic relevance (0.0 to 1.0)

Attributes

Supertypes
trait Serializable
trait Product
trait Equals
class Object
trait Matchable
class Any
Show all
Generated with