org.llm4s.agent.graph.middleware
Members list
Type members
Classlikes
A cross-cutting concern - approval, guardrails, logging, retry, rate limits - around an agent run, its model calls and its tool calls. Every hook passes through by default; override only those the concern needs.
A cross-cutting concern - approval, guardrails, logging, retry, rate limits - around an agent run, its model calls and its tool calls. Every hook passes through by default; override only those the concern needs.
Middleware run as a MiddlewareStack, ordered by registration and by runsBefore and runsAfter: the first in stack order is the outermost wrapper and runs beforeAgent first; unwinding, and afterAgent, run in reverse. A hook that throws fails the run with GraphError.MiddlewareFailed; a thrown cancellation cancels it.
wrapToolCall runs concurrently for the calls of one batch (up to RunBudgets.maxConcurrency), on task threads, so a middleware's own state must be thread-safe.
A wrapper should pass ToolOutcome.Fatal(CancelledError) through and not retry it: the call was cancelled, and a retry gets the same outcome without running the tool again. It should pass a question through too - a Left(MiddlewareAsked) from next, or a ToolOutcome.Ask - since asking again only asks again.
A middleware that asks typed questions - for review, an edit, or missing information - extends AgentMiddleware.Asking.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
- Known subtypes
-
class ApprovalMiddlewareclass ContextWindowMiddlewareclass GuardrailMiddlewareclass PermissionMiddlewareclass SemanticMemoryMiddlewareclass SkillsMiddlewareclass SummarizationMiddlewareclass TodoMiddlewareclass ToolOutputOffloadShow all
Attributes
- Companion
- trait
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
AgentMiddleware.type
Asks for approval before a tool call runs. requires returns the reason to ask, or None to let the call through; a call that is already approved always passes. A tool's own approval request is unaffected.
Asks for approval before a tool call runs. requires returns the reason to ask, or None to let the call through; a call that is already approved always passes. A tool's own approval request is unaffected.
Attributes
- Companion
- object
- Source
- ApprovalMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ApprovalMiddleware.scala
- Supertypes
- Self type
-
ApprovalMiddleware.type
What beforeAgent and afterAgent return: the text - the run's input, or its final answer - possibly changed, and an update to keys the middleware declares in its writes. beforeAgent's commits with the turn's input, so the turn's first model call sees it; afterAgent's with the turn's final answer.
What beforeAgent and afterAgent return: the text - the run's input, or its final answer - possibly changed, and an update to keys the middleware declares in its writes. beforeAgent's commits with the turn's input, so the turn's first model call sees it; afterAgent's with the turn's final answer.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
BoundaryResult.type
Prunes the messages each model call is sent to config, leaving the stored history whole. The system prompt, which reaches the model call first, is kept out of pruning and stays first; it is not counted against maxMessages or maxTokens. A tool call is never sent without its results, nor a result without its call; the current turn (from the latest user message) is never pruned - the strategy runs on the history before it, with the budget the turn leaves - and the pruned history starts with a user message, so a send may exceed the budget when the current turn alone is larger.
Prunes the messages each model call is sent to config, leaving the stored history whole. The system prompt, which reaches the model call first, is kept out of pruning and stays first; it is not counted against maxMessages or maxTokens. A tool call is never sent without its results, nor a result without its call; the current turn (from the latest user message) is never pruned - the strategy runs on the history before it, with the budget the turn leaves - and the pruned history starts with a user message, so a send may exceed the budget when the current turn alone is larger.
Attributes
- Source
- ContextWindowMiddleware.scala
- Supertypes
A guardrail refused the run's input or answer. guardrail is the first failing guardrail's name and reason every failing guardrail's error, joined with "; ". The tool loop blocks the run with it (RunResult.Failed, the thread usable), and Agent reports it as AgentStatus.Blocked.
A guardrail refused the run's input or answer. guardrail is the first failing guardrail's name and reason every failing guardrail's error, joined with "; ". The tool loop blocks the run with it (RunResult.Failed, the thread usable), and Agent reports it as AgentStatus.Blocked.
Attributes
- Source
- GuardrailBlocked.scala
- Supertypes
-
trait NonRecoverableErrortrait LLMErrortrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Runs input guardrails on the run's input (beforeAgent) and output guardrails on its final answer (afterAgent). Each list runs in order, each guardrail on the value the previous one returned, so a Fix transforms; a Warn passes. Failures are collected into one GuardrailBlocked, which blocks the run (design 4.13): it ends as a finished failure, an input block storing nothing of the turn and an output block removing it, and the thread stays usable. Agent reports it as AgentStatus.Blocked. A guardrail does not suspend.
Runs input guardrails on the run's input (beforeAgent) and output guardrails on its final answer (afterAgent). Each list runs in order, each guardrail on the value the previous one returned, so a Fix transforms; a Warn passes. Failures are collected into one GuardrailBlocked, which blocks the run (design 4.13): it ends as a finished failure, an input block storing nothing of the turn and an output block removing it, and the thread stays usable. Agent reports it as AgentStatus.Blocked. A guardrail does not suspend.
On a root agent it guards the whole agent family: its input guardrails run on every turn's query and its output guardrails on every final answer, whichever agent is active after a handoff. On a handoff target it applies only while that agent is active, inside the root's.
Attributes
- Companion
- object
- Source
- GuardrailMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- GuardrailMiddleware.scala
- Supertypes
- Self type
-
GuardrailMiddleware.type
What a reviewer is asked when a guardrail refuses a turn's query (Input) or its final answer (Output): the first failing guardrail, every failure's error, and the refused text.
What a reviewer is asked when a guardrail refuses a turn's query (Input) or its final answer (Output): the first failing guardrail, every failure's error, and the refused text.
Attributes
- Companion
- object
- Source
- GuardrailReviewMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- GuardrailReviewMiddleware.scala
- Supertypes
- Self type
-
GuardrailReview.type
Runs guardrails as GuardrailMiddleware does, but a refusal asks a reviewer instead of blocking: the turn suspends with a GuardrailReview, and the GuardrailVerdict it is answered with lets the text through, replaces it, or blocks the run as GuardrailMiddleware would have. Text the guardrails accept passes without a question, and a guardrail that transforms (Fix) still applies.
Runs guardrails as GuardrailMiddleware does, but a refusal asks a reviewer instead of blocking: the turn suspends with a GuardrailReview, and the GuardrailVerdict it is answered with lets the text through, replaces it, or blocks the run as GuardrailMiddleware would have. Text the guardrails accept passes without a question, and a guardrail that transforms (Fix) still applies.
While an input review waits, nothing of the turn is stored; while an output review waits, the answer is stored but the turn has no outcome yet. A verdict is used only for the text and phase it was asked about, whatever the refusal's reason, and the guardrails do not run on that text again once it is given, so a judge whose reason varies is asked once; if the hook runs again on different text, the guardrails run on it and the reviewer may be asked again.
Attributes
- Source
- GuardrailReviewMiddleware.scala
- Supertypes
A reviewer's verdict on a refused query or answer.
A reviewer's verdict on a refused query or answer.
Attributes
- Source
- GuardrailReviewMiddleware.scala
- Supertypes
-
trait Enumtrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
A middleware's question on its way out of a hook, from AgentMiddleware.Asking.ask: the agent loop suspends the run with it. A wrapper that sees it as next's result passes it on. Outside the loop it is an error: a question nobody can answer.
A middleware's question on its way out of a hook, from AgentMiddleware.Asking.ask: the agent loop suspends the run with it. A wrapper that sees it as next's result passes it on. Outside the loop it is an error: a question nobody can answer.
Attributes
- Source
- AgentMiddleware.scala
- Supertypes
-
trait NonRecoverableErrortrait LLMErrortrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Attributes
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
MiddlewareId.type
Middleware in the order they run, built and checked whole by MiddlewareStack.of.
Middleware in the order they run, built and checked whole by MiddlewareStack.of.
ordered(0) is the outermost wrapper: its next calls ordered(1)'s hook, and the last one's next calls the innermost function. beforeAgent runs in stack order and afterAgent in reverse; each stops at the first Left. Every hook invocation is guarded on its own: a throw becomes GraphError.MiddlewareFailed naming that middleware, and a thrown cancellation a CancelledError, restoring the interrupt flag at once, which each enclosing wrapper sees as its next's result. The innermost function is not guarded; its caller guards it.
Attributes
- Companion
- object
- Source
- MiddlewareStack.scala
- Supertypes
Attributes
- Companion
- class
- Source
- MiddlewareStack.scala
- Supertypes
- Self type
-
MiddlewareStack.type
What a middleware's beforeAgent, wrapModelCall, promptSections and afterAgent hooks see: the task's org.llm4s.agent.graph.RunContext and the committed org.llm4s.agent.graph.ThreadState the task read. A hook never sees an update made in the same superstep, its own included: an update is visible from the next one.
What a middleware's beforeAgent, wrapModelCall, promptSections and afterAgent hooks see: the task's org.llm4s.agent.graph.RunContext and the committed org.llm4s.agent.graph.ThreadState the task read. A hook never sees an update made in the same superstep, its own included: an update is visible from the next one.
The loop counts every model call it makes for the agent. A hook that calls a model itself - or hands a client to a library that does, such as a summariser, a memory manager or an LLM judge - wraps the client with accounted, so those calls are counted too, also when the task later fails (org.llm4s.agent.graph.RunEvent.RunFailed carries them). Usage that arrives without a client call is recorded with recordUsage.
Under a token budget (RunBudgets.maxTokens) every accounted call is checked before it is made, as the loop checks its own: at the budget it is refused with a Left. In wrapModelCall and promptSections the model step then ends the turn TokenLimitReached, whatever the wrapper does with the refusal; in beforeAgent and afterAgent the hook handles it as any failed call (an LLM guardrail blocks). A completion that reports no usage is charged an estimate (the request's messages in, the completion out), never zero, so the budget still advances.
Attributes
- Companion
- object
- Source
- ModelContext.scala
- Supertypes
Attributes
- Companion
- class
- Source
- ModelContext.scala
- Supertypes
- Self type
-
ModelContext.type
One model call as a model wrapper sees it: the conversation so far and the tools offered. A wrapper that rewrites the request builds the next one with withMessages or withTools.
One model call as a model wrapper sees it: the conversation so far and the tools offered. A wrapper that rewrites the request builds the next one with withMessages or withTools.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
ModelRequest.type
What a model wrapper returns: the model's completion, and an update to keys the wrapper declares in its writes, committed with the model step's task. A wrapper adds its writes with addUpdate, which keeps those of the wrappers inside it - the combined update applies innermost first - and changes the completion with withCompletion, which keeps the update. A wrapper that answers without calling next builds its own with ModelResponse(completion). Only the response a wrapper returns counts: the updates of a response it discarded, calling next again, are never committed.
What a model wrapper returns: the model's completion, and an update to keys the wrapper declares in its writes, committed with the model step's task. A wrapper adds its writes with addUpdate, which keeps those of the wrappers inside it - the combined update applies innermost first - and changes the completion with withCompletion, which keeps the update. A wrapper that answers without calling next builds its own with ModelResponse(completion). Only the response a wrapper returns counts: the updates of a response it discarded, calling next again, are never committed.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
ModelResponse.type
What a PermissionRule decides for a resource it matches.
What a PermissionRule decides for a resource it matches.
Attributes
- Source
- PermissionMiddleware.scala
- Supertypes
-
trait Enumtrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Ordered permission rules over each tool call's resources (design §5.6), checked after the call's arguments were validated and decoded, on ToolCallRequest.resources.
Ordered permission rules over each tool call's resources (design §5.6), checked after the call's arguments were validated and decoded, on ToolCallRequest.resources.
For each resource the first rule that matches it decides, and a resource no rule matches is denied. A call runs only if every resource is allowed: a denied one gives ToolOutcome.Error("Denied: <operation> <target>") (the first, in resource order); otherwise one to ask about gives NeedsApproval, unless the call is already approved. Approval is one flag per call: once a reviewer approves it - whichever middleware or tool asked - every Ask passes. An edited call is checked again on its new arguments, so a deny rule still refuses it.
Register it first, so it wraps every other middleware and a denied call is refused before anything else asks for approval of it. An Execute rule limits which commands the agent may ask to run, not what a command does: a sandbox constrains that itself.
Attributes
- Companion
- object
- Source
- PermissionMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- PermissionMiddleware.scala
- Supertypes
- Self type
-
PermissionMiddleware.type
Decides operations on targets target matches. In a PermissionMiddleware, the first matching rule wins.
Decides operations on targets target matches. In a PermissionMiddleware, the first matching rule wins.
Attributes
- Companion
- object
- Source
- PermissionMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- PermissionMiddleware.scala
- Supertypes
- Self type
-
PermissionRule.type
Where a middleware must run relative to another (AgentMiddleware.placement): a position its correctness depends on, which the stack checks rather than leaving to the documentation.
Where a middleware must run relative to another (AgentMiddleware.placement): a position its correctness depends on, which the stack checks rather than leaving to the documentation.
Attributes
- Source
- AgentMiddleware.scala
- Supertypes
-
trait Enumtrait Serializabletrait Producttrait Equalsclass Objecttrait Matchableclass AnyShow all
Content a middleware adds to the system prompt of a model request, built for the request and never stored. id is unique within one request; cacheable content is placed before the rest, so a provider's prompt cache sees a stable prefix.
Content a middleware adds to the system prompt of a model request, built for the request and never stored. id is unique within one request; cacheable content is placed before the rest, so a provider's prompt cache sees a stable prefix.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
PromptSection.type
One tool call as a tool wrapper sees it, after its arguments were validated and decoded. It is read-only: a wrapper cannot change a call's arguments. resources are what the call touches, from its decoded arguments and its tool's AgentToolSpec.withResources; never empty, since a call that declares none touches its tool's defaultResource.
One tool call as a tool wrapper sees it, after its arguments were validated and decoded. It is read-only: a wrapper cannot change a call's arguments. resources are what the call touches, from its decoded arguments and its tool's AgentToolSpec.withResources; never empty, since a call that declares none touches its tool's defaultResource.
Attributes
- Companion
- object
- Source
- AgentMiddleware.scala
- Supertypes
Attributes
- Companion
- class
- Source
- AgentMiddleware.scala
- Supertypes
- Self type
-
ToolCallRequest.type
Types
Stable identifier of an AgentMiddleware in a MiddlewareStack; must match [a-zA-Z0-9_-]{1,64}, which MiddlewareStack.of checks. Persisted in approval requests.
Stable identifier of an AgentMiddleware in a MiddlewareStack; must match [a-zA-Z0-9_-]{1,64}, which MiddlewareStack.of checks. Persisted in approval requests.
Attributes
- Source
- AgentMiddleware.scala
