The Same Context, Billed on Every Turn: How to Cut Agentic AI Token Costs

If your agent feels cheap in the prototype and terrifying on the invoice, the cause is almost always the same: an LLM is stateless, so every turn re-sends the entire context (system prompt, tool schemas, files read, and the full transcript so far) as fresh input tokens. S7 The fix is not a cheaper model. It's five techniques that stop you paying for the same tokens twice: prompt caching, semantic caching, lazy-loading tools, routing and cascading, and compaction. Stacked, published case studies and benchmarks put the savings at 60–90% with no measurable quality loss. S8S26 An unoptimized agent running 100 messages a day at ~166K input tokens can run roughly $1,000–$2,500 a month depending on model; the same workload tuned lands closer to $50–$100. S3

This is a layered problem, so this is a layered playbook. Apply them in order, caching first, because it's the highest-ROI change you can make to an API bill. S8

Why the bill grows when nothing seems to change

There is no session stored on the provider's servers. Each request is independent, so the runtime re-encodes your whole prompt every single time. S1 Turn 1 might send 5,000 input tokens. By turn 30 the model is carrying 25,000–35,000 input tokens of accumulated context on every request, the system prompt, dozens of retrieved files, and the entire prior transcript. S7 Multiply that by a loop that runs dozens of steps to finish one task, and the per-call "fraction of a cent" becomes a real number.

Two more things make it worse. Output tokens typically cost 3x to 10x more than input tokens, so a chatty agent is expensive on both ends. S2 And retry loops compound fast, a tool that errors and gets re-called drags its bloated context along for the ride. S7

Everything below is about attacking one of three levers: stop reprocessing tokens you've already paid for, stop loading tokens you don't need, and stop sending tokens to a model that's overkill for the task.

Layer 1, Prompt caching: stop paying to re-read your own prompt

Prompt caching is the single most impactful change for any app with a large, reused prefix. S10 Under the hood it's K/V caching: the provider stores the key-value tensors computed from a stable prompt prefix, then skips that computation on the next request that starts with the identical bytes. S5

The mechanics differ by provider. Anthropic uses explicit, developer-controlled breakpoints, you tag a content block with cache_control and everything up to and including it gets cached. S12 OpenAI caches automatically once a stable prefix exceeds ~1,024 tokens, billing the cached portion at 50% of the normal input rate with no code changes. S13 Google offers both implicit caching (automatic, no guaranteed savings) and explicit context caching with guaranteed discounts. S5

Anthropic's pricing is the one worth memorizing because it's the most controllable:

• Cache write: 1.25× base input for a 5-minute TTL, or 2.0× for a 1-hour TTL. S12
• Cache read: 0.10× base input, a 90% discount. S12

That read discount breaks even after roughly two reads on the 5-minute tier, or three reads on the 1-hour tier. S15 Output tokens are unaffected; caching only changes the input side. S15

Here's the minimal Anthropic setup, note that order matters, because the cache is strictly prefix-based:

{
  "system": [
    {
      "type": "text",
      "text": "You are a code reviewer for a strict TypeScript monorepo...",
      "cache_control": { "type": "ephemeral", "ttl": "1h" }
    }
  ],
  "messages": [{ "role": "user", "content": "Review this diff." }]
}

Three rules decide whether you actually get the hit:

1. Put stable content first. System instructions, tool schemas, retrieval context, and few-shot examples go at the front; the dynamic user turn goes last. S8 Anything that changes mid-prefix invalidates everything after it. S14
2. Mind the TTL default. As of March 6, 2026, Anthropic's default TTL dropped from 1 hour to 5 minutes. S11 If your traffic is bursty (say, a request every 7 minutes) you'll pay the write cost every time and never read. Set "ttl": "1h" explicitly for shared static content like a system prompt that needs to stay warm across users. S13S14
3. Watch the breakpoint limit. You get up to four breakpoints per request, and if you modify content earlier than the 20 blocks before a breakpoint, you lose the hit. Teams running long agent loops add intermediate breakpoints every ~18 blocks to keep more of the prefix cached. S14S15

The numbers justify the fuss. ProjectDiscovery's Neo agent documented a 59% cost drop from prompt caching alone, climbing past 90% on fully optimized paths. S8S14 Track your cache_read_input_tokens to input_tokens ratio in the console, aim for a hit rate above 70% on stable-prompt workloads. S2S11

Layer 2, Semantic caching: skip the model entirely on near-duplicate queries

Prompt caching and semantic caching sound similar and do completely different things. S4 Prompt caching requires a 100% exact prefix match and discounts the input you reprocess. S12 Semantic caching skips the LLM call altogether when a new query is close enough in meaning to one you've already answered.

The pattern: embed the incoming query, run a vector similarity search against recent queries, and if cosine similarity clears a threshold, typically 0.92–0.95, return the cached response instead of hitting the model. S33

q_vec = embed(user_query)
hit = vector_store.search(q_vec, k=1)
 
if hit and hit.score >= 0.93:
    return hit.cached_response          # zero LLM tokens spent
else:
    answer = call_llm(user_query)
    vector_store.upsert(q_vec, answer)
    return answer

Realistic hit rates run 25–40% for customer-facing apps and 50–60% for internal tools with repetitive queries, and your API cost drops roughly in proportion to the hit rate. S33 Common implementations use GPTCache, Redis with vector search, or pgvector. S33 For fully deterministic outputs (status checks, periodic reports, FAQ responses) plain application-layer response caching eliminates the call before you even reach the semantic layer. S32

The one caveat: set the threshold carefully. Too low and you'll return confidently wrong answers to questions you never actually answered.

Layer 3, Lazy-load tools: don't pay for a toolbox you never open

This is the silent killer in MCP-heavy setups. The more tools a model knows about, the more schema it has to carry in context on every interaction, and all of it loads before your first prompt does any work. S19 A typical enterprise MCP server with 400 tools consumes over 400,000 tokens just loading schemas, which doesn't even fit inside a 200K context window. S21

Progressive disclosure fixes this by revealing tool complexity gradually. Instead of front-loading every definition, the agent discovers tools on demand and fetches the full schema only when it's about to use one. S21 Anthropic's MCP Tool Search is the retrofit version: 50 MCP tools that previously cost ~77K tokens drop to ~8.7K with tool search active, about an 85% reduction, preserving 95% of the context window for actual work. S24

There's a quality dividend too, not just a cost one. Model accuracy degrades when it has to choose from too many tools; research puts the threshold around 20–25 tools before selection accuracy measurably declines. S21 Anthropic's internal testing found that lazy tool loading improved Opus 4's tool-selection accuracy from 49% to 74%, fewer irrelevant options means more reasoning spent on the actual task. S21

Agent Skills are the same idea at the knowledge layer. In Claude Code, only a ~1,024-character description loads per skill at startup; the full SKILL.md body loads only when the skill is relevant, and supporting files load only when referenced. S23S24 The pattern generalizes: always load high-frequency tools, and defer rare ones behind a search interface or on-demand fetch. S23

A practical host-layer rule you can implement today without waiting for protocol changes:

- Tools used > 60% of sessions   →  load eagerly
- Tools used < 10% of sessions   →  expose name + 1-line description only;
                                     hydrate full schema on first call

You'll almost always find a Pareto pattern, a handful of verbose tools that rarely get used are eating a disproportionate slice of context. S23 That's where the cheapest wins live.

Layer 4, Routing and cascading: stop sending easy work to expensive models

Most organizations use their most expensive model for every request, and that single habit drives the bulk of avoidable spend. S33 Not every task needs a frontier model. Routing dispatches each query to the cheapest model that can handle it adequately; cascading starts cheap and escalates only when the cheap model isn't confident. S29S32

A lightweight classifier, a small model or even a rules-based check, sorts requests by complexity, then routes. S33 In practice, 50–70% of enterprise requests can be handled by the cheapest tier, S33 and a well-built cascade lands around 87% cost reduction by ensuring the expensive model only sees the ~10% of queries that genuinely need it. S32

tier = classify_complexity(query)   # tiny/fast classifier
 
if tier == "simple":
    model = "haiku"     # lookups, formatting, extraction
elif tier == "standard":
    model = "sonnet"    # most code and analysis
else:
    model = "opus"      # architectural reasoning, hard debugging
 
answer = call(model, query)

You don't have to hand-roll the gateway. LiteLLM, Portkey, OpenRouter, and Bifrost all support multi-model routing and fallback out of the box, S32 and a production-grade gateway can cut costs up to 85% while holding ~95% of top-tier performance. S29 If you live in Claude Code, the built-in version is /model opusplan at session start, Opus for planning, a cheaper model for execution. S8

A related lever: don't make the model reason when it doesn't need to. Selective reasoning, only enabling chain-of-thought or higher "thinking effort" on hard queries, has been shown to cut token consumption by ~48% while improving accuracy on a reasoning benchmark, because indiscriminate reasoning wastes tokens on straightforward questions. S31

Layer 5, Compaction and clean context: pay for signal, not history

The last lever attacks accumulated history. Long sessions don't just get expensive, they get confused. As context fills, performance degrades in a way Anthropic calls "context rot": the agent starts re-reading files it already read, restating decisions, and drifting from your original constraints. S1S37 Context drift kills agents before context limits do. S9

There are three complementary moves here.

Compaction. Anthropic shipped a server-side Compaction API in beta (January 2026, for Opus 4.6 and Sonnet 4.6 and carried into Opus 4.8) that automates the summarize-and-replace loop. When input tokens cross a configurable threshold, the API summarizes the conversation so far, drops the older message blocks, and continues from the compressed summary, no client-side history juggling. S39S40 Crucially for your bill, only the new summary needs to be written as a fresh cache entry. S39

import anthropic
 
client = anthropic.Anthropic()
 
response = client.beta.messages.create(
    betas=["compact-2026-01-12"],
    model="claude-opus-4-8",
    max_tokens=4096,
    messages=messages,
    context_management={"edits": [{"type": "compact_20260112"}]},
)

Sub-agents. Instead of one agent dragging the entire context through every step, spin up separate agent instances for isolated parts of a task, so no single agent has to carry the whole thing. S1 It's a cleaner solution for complex, long-running work, and it keeps each context window focused.

Trim at ingestion, not at compression. Tool output verbosity is the primary token-budget killer. S9 Filter and truncate tool responses as they come in, keep only what the agent needs to reason about next, rather than letting junk accumulate and paying to summarize it later. S9 Removing that noise typically clears 30–70% of a context window, saving roughly the same in dollars. S4

The payoff isn't only financial. On SWE-bench Verified, a 6x context-compression approach delivered a 51.8–71.3% token-budget reduction and a 5.0–9.2% improvement in issue-resolution rates, cleaner context is both cheaper and better. S4 Guideline-based compaction methods (ACON) report 26–54% reductions in peak token usage on agentic benchmarks. S9

Watch for the drift signals as your trigger: re-reads of already-processed content, restated decisions, or the goal wording quietly changing mid-task. S9

One more free win: cap your outputs and batch your offline work

Two quick ones that don't fit a layer but pay for themselves. First, set max_tokens deliberately, output tokens are your 3x–10x cost multiplier, so an unbounded ceiling is an open tab. S2 Second, anything that doesn't need a real-time answer (overnight document processing, bulk classification, evals) belongs on a Batch API, which runs at a 50% discount. S32

And put a guardrail on the whole thing: soft budget alerts at 50% and 80% of your monthly cap, plus a hard limit that pauses processing at 100%. A single runaway loop from a coding bug can drain a budget overnight. S2

Decision checklist: which layer to reach for

Reach for prompt caching if…

• You have a large, stable system prompt, tool schema, or document context reused across calls.
• Your cache-read-to-input ratio is low and your prompts haven't changed, you may be silently on the 5-minute TTL. S11
• You're running multi-turn agent loops (the prefix barely changes between steps).

Reach for semantic caching if…

• Your traffic has meaningful query repetition (support bots, internal FAQ tools).
• Near-duplicate questions are common and exact-match caching keeps missing them.

Reach for lazy-loading if…

• You've connected more than ~20–25 tools and notice the agent picking wrong tools. S21
• A huge chunk of your context is consumed before the first user message. S19

Reach for routing/cascading if…

• You're sending every request to your most expensive model.
• Your traffic has a clear easy/hard split (most of it is easy).

Reach for compaction if…

• Sessions are long-running and hit the context window.
• The agent is showing drift, re-reading files, forgetting constraints. S1

Don't bother (yet) if…

• You're pre-product-market-fit at trivial volume, instrument first, optimize when the bill is real.
• Your prompts are unique every time with no reuse (caching won't help; focus on routing and output limits).
• Every query genuinely needs frontier-model reasoning (routing buys you little; compaction and caching still do).

Pricing, TTL defaults, and beta feature availability in this space change quickly, verify the current details in each provider's pricing page and docs before publishing.

Sources

1. S1Muhammad Attaullah Bhatti, "Why Long AI Sessions Get Expensive and Confused." Medium. https://muhammadattaullahbhatti.medium.com/why-long-ai-sessions-get-expensive-and-confused-token-burning-prompt-caching-context-windows-and-5b2258b2454d
2. S2Fastio, "AI Agent Token Cost Optimization: Complete Guide for 2026." https://fast.io/resources/ai-agent-token-cost-optimization/
3. S3Ida Silfverskiöld, "Agentic AI: How to Save on Tokens." Data Science Collective, Medium. https://medium.com/data-science-collective/agentic-ai-how-to-save-on-tokens-9a1571ac6c85
4. S4"Agentic AI: How to Save on Tokens." Towards Data Science. https://towardsdatascience.com/agentic-ai-how-to-save-on-tokens/
5. S5"Don't Break the Cache: An Evaluation of Prompt Caching for Long-Horizon Agentic Tasks." arXiv. https://arxiv.org/pdf/2601.06007
6. S6NVIDIA Technical Blog, "Building for the Rising Complexity of Agentic Systems with Extreme Co-Design." https://developer.nvidia.com/blog/building-for-the-rising-complexity-of-agentic-systems-with-extreme-co-design/
7. S7Vantage, "The Hidden Cost Driver in Agentic Coding Sessions in 2026." https://www.vantage.sh/blog/agentic-coding-costs
8. S8ProgramStrategyHQ, "Techniques to Reduce AI Token Usage: The 2026 Playbook." https://www.programstrategyhq.com/post/techniques-to-reduce-ai-token-usage-the-2026-playbook-for-cutting-costs-without-losing-quality
9. S9Zylos Research, "AI Agent Context Compression: Strategies for Long-Running Sessions." https://zylos.ai/research/2026-02-28-ai-agent-context-compression-strategies/
10. S10Finout, "Anthropic API Pricing in 2026: Complete Guide." https://www.finout.io/blog/anthropic-api-pricing
11. S11DEV Community, "Anthropic Silently Dropped Prompt Cache TTL from 1 Hour to 5 Minutes." https://dev.to/whoffagents/anthropic-silently-dropped-prompt-cache-ttl-from-1-hour-to-5-minutes-16ao
12. S12Claude API Docs, "Prompt caching." https://platform.claude.com/docs/en/build-with-claude/prompt-caching
13. S13Technspire, "Prompt Caching in 2026: Anthropic, OpenAI, Azure Compared." https://technspire.com/en/blog/prompt-caching-2026-real-cost-wins
14. S14ProjectDiscovery, "How We Cut LLM Costs by 59% With Prompt Caching." https://projectdiscovery.io/blog/how-we-cut-llm-cost-with-prompt-caching
15. S15DEV Community, "Prompt Caching With the Claude API: A Practical Guide." https://dev.to/thegdsks/prompt-caching-with-the-claude-api-a-practical-guide-14ce
16. S16Solo.io, "Reduce MCP Token Usage with Agentgateway Progressive Disclosure." https://www.solo.io/blog/keeping-context-and-tokens-low-with-progressive-disclosure-in-agentgateway
17. S17Matthew Kruczek, "Progressive Disclosure MCP: 85x Token Savings Benchmark." https://matthewkruczek.ai/blog/progressive-disclosure-mcp-servers.html
18. S18Layered System, "MCP Tool Schema Bloat: The Hidden Token Tax (and How to Fix It)." https://layered.dev/mcp-tool-schema-bloat-the-hidden-token-tax-and-how-to-fix-it/
19. S19Software Thug, "Claude Code MCP Tool Search: How Lazy Loading Cut Token Usage by 85%." https://www.softwarethug.com/posts/claude-code-mcp-tool-search-lazy-loading/
20. S20PremAI, "LLM Cost Optimization: 8 Strategies That Cut API Spend by 80% (2026 Guide)." https://blog.premai.io/llm-cost-optimization-8-strategies-that-cut-api-spend-by-80-2026-guide/
21. S21Maxim, "Top 5 LLM Routing Techniques." https://www.getmaxim.ai/articles/top-5-llm-routing-techniques/
22. S22"When to Reason: Semantic Router for vLLM." arXiv. https://arxiv.org/pdf/2510.08731
23. S23Zylos Research, "AI Agent Cost Optimization: Token Economics and FinOps in Production." https://zylos.ai/research/2026-02-19-ai-agent-cost-optimization-token-economics
24. S24TechCloudPro, "How to Cut Enterprise LLM Costs by 50%: Caching, Routing, and Infrastructure Strategies." https://techcloudpro.com/blog/enterprise-llm-cost-optimization/
25. S25InfoQ, "Claude Opus 4.6 Introduces Adaptive Reasoning and Context Compaction for Long-Running Agents." https://www.infoq.com/news/2026/03/opus-4-6-context-compaction/
26. S26Claude API Docs, "Compaction." https://platform.claude.com/docs/en/build-with-claude/compaction
27. S27Claude Lab, "Context Compaction API Complete Guide." https://claudelab.net/en/articles/api-sdk/compaction-api-context-management

Written by

Syed Moinuddin

Full Stack Engineer writing about AI tooling, agentic systems, and the frontend/backend craft. June 2026.