Limits on one run; each start, recover or resume brings its own. Every value is validated on the way in: apply and the with* setters throw IllegalArgumentException for a non-positive one, and RunBudgets.of returns it as a ValidationError.
maxTokens is the input and output tokens the run may use, counted from the thread's usage when the run was admitted: before each model call - the loop's own and every one a middleware makes through next or ModelContext.accounted - the run's tokens so far are compared with it, and at or above it the turn ends TokenLimitReached without calling the model. A model call in flight is never interrupted, so a run can overshoot the budget by one call. A completion without usage is charged an estimate of its whole request and answer. A retrying client (ReliableClient) is one call: its failed attempts carry no usage and are not counted, so set maxTokens with retries in mind. None, the default, sets no token budget. A plain graph's nodes never read it.
Attributes
- Companion
- object
- Source
- RunConfig.scala
- Graph
-
- Supertypes
