How to Cut Token Cost for a Multi-Tenant AI Agent¶

To cut token cost, multi-tenant AI agent deployments have to fight three inefficiencies at once — a leaky shared cache, unbounded tool-loop transcripts, and the full set of tool schemas resent on every call — because one agent serving many customers multiplies each of them per tenant. A waste that is annoying for a single-user demo becomes the dominant line on your bill when a thousand tenants hit the same code path. This post is the hub that connects the three cost levers Promptise Foundry ships as first-class, composable primitives on one build_agent() call — a per-tenant semantic cache, a context_scope="ledger" transcript, and optimize_tools — and then shows you how to prove the savings with built-in observability rather than trust a marketing number.

Not legal or compliance advice

The information here is general technical information, not legal, regulatory, or compliance advice. Descriptions of any law, regulation, or standard (such as the GDPR, the EU AI Act, HIPAA, SOC 2, or PCI DSS) are simplified and may be incomplete, out of date, or inaccurate, and requirements vary by jurisdiction and situation. Promptise Foundry makes no warranty as to the accuracy or completeness of this content and is not responsible for how you use or rely on it. Using Promptise does not by itself make you or your product compliant with any law or standard. Consult a qualified lawyer or compliance professional before acting on anything here.

Why one agent for many tenants multiplies every wasted token¶

Multi-tenancy is a cost amplifier. The same three token sinks that a hobby agent can ignore all compound linearly with your customer count.

The tool-definition tax, paid per tenant, per call. Every MCP tool's name, description, and full JSON Schema is resent to the model on every invocation. With 20–50 tools that is 5,000–15,000+ tokens before the conversation even starts. One tenant pays it once; a thousand tenants pay it a thousand times a minute.
Unbounded tool-loop transcripts on deep tasks. A naive tool-calling loop feeds the model the entire growing conversation on every turn. By the twentieth call it is re-reading a wall of its own past work — re-fetching facts it already has and paying for thousands of redundant tokens. Now run that loop concurrently for hundreds of tenants.
A cache that either doesn't isolate or doesn't exist. A response cache is the single biggest cost lever on repetitive traffic — support agents, doc Q&A, and internal tools ask near-identical questions constantly. But in a multi-tenant system a naive shared cache is a data-leak vector, and a cache that fails to isolate by tenant is worse than no cache at all.

The trap is that each of these has a well-known fix, but in most stacks the three fixes are three separate libraries you assemble, secure, and keep in sync yourself. The goal of this guide is to treat multi-tenant agent cost optimization as one configuration surface, not a scavenger hunt.

The three cost levers, configured on one build_agent() call¶

Promptise makes each lever a parameter on the same constructor. You do not wire three subsystems together — you turn three flags on.

Lever 1 — optimize_tools: stop resending schemas you don't need. Tool optimization strips verbose per-field descriptions from schemas (the model still sees names, types, and required status) and, at the semantic level, embeds every tool description once at build time and selects only the top-K relevant tools per query. Instead of shipping all 50 tools on every call, the model sees the handful the current question actually needs, plus a request_more_tools fallback so it can self-recover if the selection missed one. The published saving is 40–70% fewer tool-definition tokens; the full mechanism and preset levels are in the tool optimization guide.

agent = await build_agent(
    servers=servers,
    model="openai:gpt-5-mini",
    optimize_tools="semantic",  # per-query tool selection, biggest saving
)

Lever 2 — a per-tenant semantic cache: serve repeats without a model call. SemanticCache matches by meaning, so "What's the monthly price of Pro?" hits the entry stored for "How much does the Pro plan cost per month?" — a published 30–50% cost cut on repetitive traffic. Critically, its default scope is per_user, and with a tenant-qualified CallerContext the cache key becomes an injective, tenant-prefixed hash: two tenants with the same user_id can never share a partition, and the similarity search only ever runs inside the caller's own partition. There is no code path where an embedding is compared across tenants, and the absence of identity disables caching rather than caching globally. That fail-closed isolation is the whole subject of Can a Paraphrase Leak Another Tenant's Cached Answer?; the configuration reference lives in the semantic cache docs.

Lever 3 — context_scope="ledger": bound the transcript without a summarizer. For deep tool chains, the managed pattern runs a single tool-using node whose view of history is a compact, deduplicated ledger — one line per tool(args) = result, last value wins — placed right before the model's turn. A fact fetched at turn 3 is still one line at turn 30 instead of scrolling out of a window, and repeated calls are cache-served with no extra model round-trip. The decision table for full vs scoped vs ledger is in the context lifecycle guide.

Because all three are parameters on one function, they compose — the cache check runs before tool selection, tool selection runs before the LLM call, and the ledger bounds the transcript within the loop. You configure the whole cost stack in one place.

What other frameworks do today¶

Be fair: the mature frameworks are not asleep on cost. Each ships some of these pieces. The precise gap is that none combine tenant-isolated LLM caching, deep-loop context bounding, and one-flag per-query tool selection as first-class, composable primitives on a single agent constructor — so you assemble and secure the set yourself.

LangChain / LangGraph ship all three pieces, separately. RedisSemanticCache (and the GPTCache wrapper it exposes) matches by embedding similarity keyed on the prompt embedding plus the model string — there is no caller or tenant dimension in that key, so per-tenant isolation means running a separate cache instance or Redis prefix per tenant that you wire and secure yourself. Context bounding is trim_messages (a genuinely useful utility that evicts by token or message count — positional, so on a deep loop it can drop the exact tool(args)=result still in play) or a summarization pre-model hook (which preserves meaning but spends an extra LLM call each time it fires). Tool selection is the separate langgraph-bigtool library: a real, well-built add-on that stores tools in a registry and retrieves relevant ones by semantic search — but it is a package you install and wire into the graph, not a flag on the constructor. Three good pieces, three integration-and-hardening jobs.
LlamaIndex bounds transcript growth with a token_limit on its memory buffer — so it is not true that it has no growth control. The delta is that it truncates oldest content to stay under the limit rather than collapsing duplicate tool calls, and its cache story runs through the same GPTCache shape with isolation as an opt-in you build.
CrewAI ships a CacheHandler that caches tool outputs keyed on tool name plus arguments — a per-crew execution cache with no tenant or user dimension — and feeds the full running transcript back to the model by default on a standard loop.
AutoGen and Pydantic AI leave most token cost to the developer. AutoGen's Cache is an exact-match key-value store keyed on the serialized request plus a manual seed; callers are isolated only if you hand them different seeds, and it is not similarity-based. Pydantic AI gives you clean hooks but ships no built-in semantic response cache or deduplicating tool ledger.

The honest summary: these are real, shipped features, and every one of them can be made multi-tenant by hand. Promptise's edge is not "we have cost control and they don't." It is that Promptise makes the per-tenant semantic cache, the context_scope ledger, and optimize_tools structural, default-on primitives on one build_agent() call — isolated by default and fail-closed — rather than three add-ons you integrate and then have to keep secure as your tenant list grows.

Prove the savings with built-in observability (runnable)¶

Never trust a percentage you didn't measure. Turn on observe=True and every LLM turn, tool call, and token count is captured automatically, so you can diff a plain agent against the optimized one on your traffic. This is a complete script — set OPENAI_API_KEY, point servers at any MCP server, and run it:

import asyncio

from promptise import build_agent, SemanticCache, CallerContext
from promptise.config import HTTPServerSpec


async def main():
    # All three cost levers + observability on one build_agent() call.
    agent = await build_agent(
        model="openai:gpt-5-mini",
        servers={"tools": HTTPServerSpec(url="http://localhost:8000/mcp")},
        cache=SemanticCache(),        # per_user scope by default → tenant-isolated
        optimize_tools="semantic",    # per-query tool selection, 40-70% fewer tool tokens
        agent_pattern="managed",      # deep tool loop runs with context_scope="ledger"
        observe=True,                 # capture token counts to prove the drop
        instructions=(
            "Answer by calling tools. A deduplicated ledger of the facts you "
            "already gathered is provided every turn — consult it and never "
            "re-fetch a fact you already have."
        ),
        max_agent_iterations=30,
    )

    # Same question from two different tenants — each gets its own cache partition.
    for tenant in ("acme", "globex"):
        caller = CallerContext(user_id="support-bot", tenant_id=tenant)
        result = await agent.ainvoke(
            {"messages": [{"role": "user",
                           "content": "How many seats are left on our current plan?"}]},
            caller=caller,
        )
        print(tenant, "->", result["messages"][-1].content)

    # Measured, not guessed: token totals across the run.
    print(agent.get_stats())
    await agent.shutdown()


asyncio.run(main())

Run it twice — once with cache, optimize_tools, and agent_pattern removed, once as written — and compare agent.get_stats(). The tool-definition tokens drop with optimize_tools, the transcript stops ballooning under the ledger (the head-to-head against LangGraph's positional trimming is in Why trim_messages Drops Facts Your Agent Still Needs), and repeated tenant queries return from the cache with zero LLM tokens on the hit. Because each CallerContext carries a tenant_id, the cache partitions are disjoint — acme never sees globex's cached answer even if they ask the exact same paraphrase.

The multi-tenant cost checklist¶

Work these in order and measure after each — layering cheap-and-safe first, then the deeper levers where the chain actually gets long.

Turn on optimize_tools ("semantic" if you have 20+ tools, "minimal" for a safe, no-behavior-change start). This is pure upside on every call.
Add SemanticCache() and always pass CallerContext(user_id=..., tenant_id=...). Keep the default per_user scope. Remember: no caller means no caching — that's the fail-closed guarantee, not a bug.
Switch your deepest tool chain to agent_pattern="managed" (or set context_scope="ledger" on the node) so the transcript stays bounded as the task gets deep.
Turn on observe=True and diff agent.get_stats() before and after each lever, on your real traffic. Ship the numbers you measured.
Wire GDPR cleanup once: await cache.purge_user(user_id, tenant_id=tenant) removes exactly one tenant's cached entries.

Frequently asked questions¶

Which lever should I turn on first to cut token cost for a multi-tenant AI agent?¶

Start with optimize_tools — it reduces the tool-definition tokens paid on every call with no behavioral change, so it is the cheapest win. Then add the per-tenant SemanticCache for repetitive traffic (the biggest saver on support and FAQ workloads), and finally reach for context_scope="ledger" on whichever tool chain actually runs deep. Measure after each with observe=True.

By default it is not shared — per_user scope isolates every partition, and a tenant_id on the CallerContext makes the key tenant-qualified so two tenants with the same user_id can never collide. The similarity search only runs inside the caller's own partition, and a missing identity disables caching rather than serving globally. The full leak analysis is in Can a Paraphrase Leak Another Tenant's Cached Answer?.

How is context_scope="ledger" different from LangGraph's trim_messages?¶

trim_messages evicts by token or message count — positional, so it can drop the exact tool(args)=result you still need on a deep loop and force a re-fetch. The ledger evicts duplicates instead: one line per unique (tool, args), last value wins, so a fact fetched early survives to aggregation time while repeats collapse — and repeated calls are cache-served with no extra model call. The side-by-side is in Why trim_messages Drops Facts Your Agent Still Needs.

Do these levers make the agent's answers less accurate?¶

Schema minification keeps field names, types, and required status — most models infer purpose from well-named parameters, and preserve_tools exempts any ambiguous ones. The ledger is an efficiency primitive: it bounds token growth at equal accuracy on long chains but does not, by itself, make a weak model aggregate facts more correctly. Be honest about which problem you have.

Can I measure the savings without paying for a benchmark run?¶

Yes — observe=True captures prompt, completion, and total token counts automatically, and agent.get_stats() returns them as a dict. Diff a plain agent against the optimized one on your own traffic to get a real number instead of a published range.

Next steps¶

Work the checklist and layer each lever on build_agent() — start with per_user cache and optimize_tools, then turn on observability to measure the drop before you add the ledger. Read the semantic cache docs for scope and GDPR details, the tool optimization guide for the preset levels, and the context lifecycle guide for the full context_scope decision table. Then go deeper on the two levers that most repay attention in a multi-tenant system: keeping the cache leak-proof and bounding the transcript without dropping live facts.