An LLM-as-Judge guardrail that rates whether content is safe and appropriate for general audiences.
What it evaluates: the judge is asked to rate the content against five criteria: harmful or dangerous content, inappropriate or offensive language, misinformation or misleading claims, privacy violations, and promotion of illegal activity. It is told that 1.0 means completely safe, 0.5 means minor concerns but nothing harmful, and 0.0 means clearly harmful, dangerous or inappropriate. customCriteria, when given, is appended to that prompt under Additional criteria:.
When to use it: when whether something is acceptable depends on context. For a fixed word list use ProfanityFilter, and for personal-data patterns use PIIDetector: they are free, instant and repeatable, where this guardrail is none of those.
Cost and side: every validation makes one extra llmClient.complete call, so it does not suit latency-sensitive paths, and the content is sent to the provider of llmClient. It is an output guardrail only: it judges the agent's final answer. The scoring rules (the judge must answer with one number from 0 to 1: a reply on any other scale, such as 85 or 8/10, is refused as unreadable and fails the guardrail) and the other limits are described on org.llm4s.agent.guardrails.LLMGuardrail.
Failure:Left with a org.llm4s.error.ValidationError on field output, for example LLM judge score (0.60) below threshold (0.80) for LLMSafetyGuardrail. It does not say which criterion failed. An unreadable reply or a failing llmClient is a Left too, never a pass.
Value parameters
customCriteria
extra criteria appended to the prompt verbatim under Additional criteria:; None keeps the five built-in criteria only
llmClient
the client that makes the judge call; it can be the agent's own or a separate model
threshold
the lowest score that passes (a score equal to it passes), between 0.0 and 1.0 or validate fails; the default 0.8 is higher than the base default of 0.7, as the original comment put it, "higher for safety"
Attributes
Example
import org.llm4s.agent.Agent
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.builtin.LLMSafetyGuardrail
val guardrail = LLMSafetyGuardrail(client)
val agent = Agent
.builder("assistant", client)
.withMiddleware(new GuardrailMiddleware(Nil, Seq(guardrail)))
.build()
The options of the judge call. The default is temperature 0.0, which makes the judge as repeatable as the model allows, and a maximum of 10 tokens, enough for a bare number. Override to give a judge model more room or to set other options.
The options of the judge call. The default is temperature 0.0, which makes the judge as repeatable as the model allows, and a maximum of 10 tokens, enough for a bare number. Override to give a judge model more room or to set other options.
Judges value with one LLM call and compares the score with threshold.
Judges value with one LLM call and compares the score with threshold.
Value parameters
value
the text to judge, typically the agent's final answer
Attributes
Returns
Right(value) unchanged when the score reaches the threshold; otherwise Left: a org.llm4s.error.ValidationError on field threshold when the threshold is not between 0.0 and 1.0 (no judge call is made), on field output when the score is too low, on field llm_response when the reply has no readable score, or the client's own error when the call fails
The evaluation criteria in natural language. It is sent after Evaluation criteria: and before the content, which goes between triple quotes. The fixed system message already asks for a bare number between 0 and 1; say in the criteria what 0 and 1 mean, because a reply on any other scale is refused as unreadable and fails the guardrail (see the scoring notes on the trait).
The evaluation criteria in natural language. It is sent after Evaluation criteria: and before the content, which goes between triple quotes. The fixed system message already asks for a bare number between 0 and 1; say in the criteria what 0 and 1 mean, because a reply on any other scale is refused as unreadable and fails the guardrail (see the scoring notes on the trait).
Attributes
Example
"Rate if this response is professional in tone. Return only a number between 0 and 1."
The client that makes the judge call. It can be the agent's own client or a different one; a separate, cheaper model is a common choice. Every validation makes one blocking complete call on it.
The client that makes the judge call. It can be the agent's own client or a different one; a separate, cheaper model is a common choice. Every validation makes one blocking complete call on it.
The lowest score that passes (a score equal to it passes). The default is 0.7. It must be between 0.0 and 1.0, both included: validate returns a org.llm4s.error.ValidationError on field threshold, without calling the judge, for a value outside that range (including infinities) or NaN. It is checked when validate runs, not when the guardrail is built, so the constructors and factories keep their signatures and a subclass that overrides it is checked too. A threshold of 0.0 passes every reply that parses as a score. It is not a probability or a confidence: it is a cut-off on whatever number the judge returns.
The lowest score that passes (a score equal to it passes). The default is 0.7. It must be between 0.0 and 1.0, both included: validate returns a org.llm4s.error.ValidationError on field threshold, without calling the judge, for a value outside that range (including infinities) or NaN. It is checked when validate runs, not when the guardrail is built, so the constructors and factories keep their signatures and a subclass that overrides it is checked too. A threshold of 0.0 passes every reply that parses as a score. It is not a probability or a confidence: it is a cut-off on whatever number the judge returns.