ModelContext

org.llm4s.agent.graph.middleware.ModelContext
See theModelContext companion object
@Experimental
final class ModelContext

What a middleware's beforeAgent, wrapModelCall, promptSections and afterAgent hooks see: the task's org.llm4s.agent.graph.RunContext and the committed org.llm4s.agent.graph.ThreadState the task read. A hook never sees an update made in the same superstep, its own included: an update is visible from the next one.

The loop counts every model call it makes for the agent. A hook that calls a model itself - or hands a client to a library that does, such as a summariser, a memory manager or an LLM judge - wraps the client with accounted, so those calls are counted too, also when the task later fails (org.llm4s.agent.graph.RunEvent.RunFailed carries them). Usage that arrives without a client call is recorded with recordUsage.

Under a token budget (RunBudgets.maxTokens) every accounted call is checked before it is made, as the loop checks its own: at the budget it is refused with a Left. In wrapModelCall and promptSections the model step then ends the turn TokenLimitReached, whatever the wrapper does with the refusal; in beforeAgent and afterAgent the hook handles it as any failed call (an LLM guardrail blocks). A completion that reports no usage is charged an estimate (the request's messages in, the completion out), never zero, so the budget still advances.

Attributes

Companion
object
Graph
Supertypes
class Object
trait Matchable
class Any

Members list

Value members

Concrete methods

def accounted(client: LLMClient): LLMClient

client, with the usage of every complete and streamComplete through it - and of completeStructured, which completes through complete - recorded in this task's usage and announced with a durable ModelCallCompleted (attempts 0). At the run's token budget a call is refused, not made: it returns Left, and a model step ends the turn TokenLimitReached. close does not close client: the hook that owns it does.

client, with the usage of every complete and streamComplete through it - and of completeStructured, which completes through complete - recorded in this task's usage and announced with a durable ModelCallCompleted (attempts 0). At the run's token budget a call is refused, not made: it returns Left, and a model step ends the turn TokenLimitReached. close does not close client: the hook that owns it does.

Attributes

def recordUsage(model: String, usage: TokenUsage, estimatedCost: Option[Double]): Unit

Records usage that arrived without a client call, as one request to model.

Records usage that arrived without a client call, as one request to model.

Attributes

Concrete fields