An LLM-as-Judge guardrail that rates the overall quality of a response to a given query.
What it evaluates: the judge is shown the original query and asked to rate the response on five points: relevance (does it address the query), helpfulness, completeness, clarity, and apparent accuracy. It is told that 1.0 means an excellent, comprehensive response, 0.5 an adequate but incomplete one, and 0.0 an irrelevant or unhelpful one. "Accuracy" here is the judge's impression: nothing is checked against a source (use LLMFactualityGuardrail for that).
When to use it: for a subjective check that the answer is on topic and useful. For size limits use LengthCheck, and for a required format use RegexValidator or JSONValidator: they are free, instant and repeatable.
Cost and side: every validation makes one extra llmClient.complete call, and the response goes to the provider of llmClient. It is an output guardrail only. The scoring rules and the other limits are described on org.llm4s.agent.guardrails.LLMGuardrail.
One query per instance:originalQuery is fixed when the guardrail is built, and it is placed in the prompt inside double quotes without escaping. A guardrail built once for an agent judges every later answer against that same query, so build a new one per query when answers should be judged against the question that produced them. Its description embeds the first 50 characters of the query followed by ....
Failure:Left with a org.llm4s.error.ValidationError on field output, for example LLM judge score (0.55) below threshold (0.70) for LLMQualityGuardrail. It does not say which of the five points fell short. An unreadable reply or a failing llmClient is a Left too, never a pass.
Value parameters
llmClient
the client that makes the judge call; it can be the agent's own or a separate model
originalQuery
the query the response is judged against
threshold
the lowest score that passes (a score equal to it passes), between 0.0 and 1.0 or validate fails; default 0.7
Attributes
Example
import org.llm4s.agent.Agent
import org.llm4s.agent.graph.middleware.GuardrailMiddleware
import org.llm4s.agent.guardrails.builtin.LLMQualityGuardrail
val guardrail = LLMQualityGuardrail(client, "What is Scala?")
val agent = Agent
.builder("assistant", client)
.withMiddleware(new GuardrailMiddleware(Nil, Seq(guardrail)))
.build()
The options of the judge call. The default is temperature 0.0, which makes the judge as repeatable as the model allows, and a maximum of 10 tokens, enough for a bare number. Override to give a judge model more room or to set other options.
The options of the judge call. The default is temperature 0.0, which makes the judge as repeatable as the model allows, and a maximum of 10 tokens, enough for a bare number. Override to give a judge model more room or to set other options.
Judges value with one LLM call and compares the score with threshold.
Judges value with one LLM call and compares the score with threshold.
Value parameters
value
the text to judge, typically the agent's final answer
Attributes
Returns
Right(value) unchanged when the score reaches the threshold; otherwise Left: a org.llm4s.error.ValidationError on field threshold when the threshold is not between 0.0 and 1.0 (no judge call is made), on field output when the score is too low, on field llm_response when the reply has no readable score, or the client's own error when the call fails
The evaluation criteria in natural language. It is sent after Evaluation criteria: and before the content, which goes between triple quotes. The fixed system message already asks for a bare number between 0 and 1; say in the criteria what 0 and 1 mean, because a reply on any other scale is refused as unreadable and fails the guardrail (see the scoring notes on the trait).
The evaluation criteria in natural language. It is sent after Evaluation criteria: and before the content, which goes between triple quotes. The fixed system message already asks for a bare number between 0 and 1; say in the criteria what 0 and 1 mean, because a reply on any other scale is refused as unreadable and fails the guardrail (see the scoring notes on the trait).
Attributes
Example
"Rate if this response is professional in tone. Return only a number between 0 and 1."
The client that makes the judge call. It can be the agent's own client or a different one; a separate, cheaper model is a common choice. Every validation makes one blocking complete call on it.
The client that makes the judge call. It can be the agent's own client or a different one; a separate, cheaper model is a common choice. Every validation makes one blocking complete call on it.
The lowest score that passes (a score equal to it passes). The default is 0.7. It must be between 0.0 and 1.0, both included: validate returns a org.llm4s.error.ValidationError on field threshold, without calling the judge, for a value outside that range (including infinities) or NaN. It is checked when validate runs, not when the guardrail is built, so the constructors and factories keep their signatures and a subclass that overrides it is checked too. A threshold of 0.0 passes every reply that parses as a score. It is not a probability or a confidence: it is a cut-off on whatever number the judge returns.
The lowest score that passes (a score equal to it passes). The default is 0.7. It must be between 0.0 and 1.0, both included: validate returns a org.llm4s.error.ValidationError on field threshold, without calling the judge, for a value outside that range (including infinities) or NaN. It is checked when validate runs, not when the guardrail is built, so the constructors and factories keep their signatures and a subclass that overrides it is checked too. A threshold of 0.0 passes every reply that parses as a score. It is not a probability or a confidence: it is a cut-off on whatever number the judge returns.