Every agent turn spends on two meters at once, model tokens and API credits, and the user pays a third in seconds. Make all three visible, stop paying twice for the same answer, and give each conversation a budget it cannot exceed.
A single tool call looks cheap: a credit or two, a few hundred milliseconds. An agent multiplies it in ways a normal program does not. It decides how many calls to make. It sends the whole conversation, tool definitions included, back to the model on every turn. And it waits for a model round trip between each step. A question that needed two lookups can turn into eight turns, fifteen calls and twelve seconds if nothing is watching.
This lesson makes the costs visible and then puts limits on them. None of it requires a cleverer model; all of it lives in the tool layer and loop you have already built.
Three meters on every conversation
Model tokens. Each request in the loop is billed for its input (system prompt, tool definitions, every prior message and tool result) and its output. Because the history is resent, a long tool result is paid for on every later turn, not once.
API credits. Each successful APIVerve call costs a fixed number of credits, set per API. You can see it on each marketplace page, in the tokenUse field of the API's definition, and in the description of every tool the APIVerve MCP server exposes, which ends with the cost. The tools used in this course:
A twenty-to-one spread inside one small tool set is normal. An agent that validates every address mentioned in a long email thread spends more than one that answers a hundred unit questions.
Seconds. Latency in an agent is mostly round trips. Each turn is one model request plus whatever tools run before the next. Tool calls that run one after another add up; a turn that could have made three calls at once but made them over three turns adds two whole model requests.
Check yourself
A tool returns a 6 KB JSON response on turn 2 of a 6-turn conversation. Roughly how many times is that response billed as model input tokens?
Cache tool results
A cached result costs no credits and returns in microseconds. The question for each tool is how long its answer stays true, and the answer comes from the data, not from the tool's name.
What the tool returns
Cache for
Examples
A pure function of its input
Indefinitely
100 lb in kg; days between two dates
A historical value with a fixed date
Indefinitely
The USD to EUR close on 2025-06-02
Reference lookups
Hours to days
City to timezone
Live values
Minutes
Current exchange rate
Validation, current time
Do not cache, or seconds at most
Email deliverability, local time
Two of those rows are worth seeing. Unit and date arithmetic never change:
A past date returns that day's close (0.879472) with lastUpdated null. It will never change.
The same tool can therefore need two policies. Key the cache on the arguments and choose the TTL from them: no date means live, cache for minutes; a past date means immutable, cache forever.
There is also a cheaper cache than any of these: the current conversation. Models often repeat a call they already made a few turns earlier. A per-conversation memo keyed on tool name and arguments catches that for free, even for tools you would never cache across users.
# Python 3.9+. A shared TTL cache plus a per-conversation memo.import json
import math
import time
_shared = {}
TTL = {
"convert_units": lambda a: math.inf,
"days_between": lambda a: math.inf,
"get_rate": lambda a: math.inf if a.get("date") else300,
"get_local_time": lambda a: 0,
"check_email": lambda a: 0,
}
def key_of(name, args):
norm = sorted((k, str(v).strip().upper()) for k, v in args.items())
return f"{name}:{json.dumps(norm)}"def make_cached_runner(tools):
memo = {} # lives for one conversationdef run(name, args):
key = key_of(name, args)
if key in memo:
return memo[key], "conversation"
hit = _shared.get(key)
if hit and hit["expires"] > time.time():
return hit["value"], "shared"
value = tools[name]["run"](args) # exceptions propagate and are not cached
memo[key] = value
ttl = TTL.get(name, lambda a: 0)(args)
if ttl > 0:
_shared[key] = {"value": value, "expires": time.time() + ttl}
return value, Falsereturn run
The key normalisation upper-cases every value, which is right for currency codes and unit symbols and wrong for anything case-sensitive. Normalise per tool when that matters; the principle from the Fundamentals caching lesson holds: only merge inputs the API really treats as equal.
Run independent calls in parallel
When a user asks "What is 250 USD in yen, and what time is it in Tokyo?", the two lookups do not depend on each other. Current models will usually request both in one turn as two tool_use blocks. Your loop should run them concurrently and return all results together in the next user message. Splitting results across several messages works, but it teaches the model that parallel requests are not handled and it tends to stop making them.
Parallel is not always right. Turn it off (in the Anthropic API, tool_choice: { type: 'auto', disable_parallel_tool_use: true }; OpenAI has parallel_tool_calls: false) when calls have side effects that must happen in order, or when a single turn could fan out into more expensive calls than you want to commit to at once. Also keep the APIVerve rate limit in mind: a fan-out of twenty calls in one turn on a plan with a low per-minute limit will trip 429s that your retry logic then has to absorb.
Put a budget on each conversation
Caching lowers the average. A budget caps the worst case, which is what protects you from a looping agent or a user pasting a thousand-address list. Track credits per conversation in your tool layer, check before each call, and refuse with a clear tool error when the budget would be exceeded. The model then answers with what it has instead of discovering the limit by failure.
budget.py
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
from tool_errors import ToolError
COST = {"convert_units": 1, "days_between": 1, "get_rate": 2, "get_local_time": 2, "check_email": 20}
PER_TOOL_CAP = {"check_email": 3}
class Budget:
def __init__(self, credits=60):
self.left = credits
self.calls = {}
def charge(self, name):
cost = COST.get(name, 5)
if self.calls.get(name, 0) >= PER_TOOL_CAP.get(name, 999):
raise ToolError(f"{name} limit reached for this conversation. Answer with the results you already have.")
if cost > self.left:
raise ToolError("Lookup budget for this conversation is used up. Answer with what you have and say what is missing.")
self.left -= cost
self.calls[name] = self.calls.get(name, 0) + 1
Call budget.charge(name) only on a cache miss, after the cache check and before the real call, and refund it if the call fails, because failed calls are not billed. A per-tool cap is often more useful than the total: three email validations per conversation is plenty for a support bot and stops the bulk-paste case at 60 credits instead of 20,000. Set the numbers from real transcripts, not guesses.
For the account as a whole, the response headers give you the real figures: x-api-credits-used and x-api-remaining-credits on every response. Log them, alert well before zero, and pair the agent's sub-key with a per-key rate limit if your plan offers one.
Check yourself
Users sometimes paste long lists of email addresses into your support agent, which validates each one at 20 credits. Which control stops the cost spike with the least effect on normal conversations?
Answer the check above to continue.
Measure every turn
You cannot tune what you have not measured, and agent costs are surprising often enough that guessing is a mistake. Log one line per tool call and one per model request, and roll them up per conversation:
The model fields come from the usage object on every model response. The tool fields come from your runner: time each call, record the cost and whether it was cached. A week of these lines answers the questions that matter: which tool is most of the spend, how often the memo hits, how many turns the median conversation takes, and whether time goes to tools or to the model.
Two optimisations come straight out of that data. If input tokens dominate, shrink tool outputs and keep the system prompt and tool list identical across requests, in a fixed order, so the model provider's prompt caching can reuse that prefix; cache_read_input_tokens tells you whether it does. If wall time dominates and tools are a small slice, the fix is fewer turns: better tool descriptions so the model gets it right first time, and parallel calls.
You now have an agent that answers from live data, sees only the tools its job needs, recovers from the errors it can fix, halts on the ones it cannot, and runs inside a known budget. The next course puts those agents to work on multi-step jobs: chaining tools, planning, and producing output you can trust without reading every transcript. Continue with Build Agent Workflows.
Finished the lesson?Mark it complete to track your progress toward the certification.
