The 1M-token context window: when you should actually use it

Long context is mainstream. Gemini 2.5 Pro at 2M tokens, Claude Sonnet's 1M-token tier, GPT-4o at 200k, plus a handful of niche models pushing 10M. The marketing implies you should be using it. The math says mostly you shouldn't.

What "1M tokens" actually means in dollars

Filling a 1M-token input on Claude Sonnet 4.5 at $3/M input costs $3.00 per request. Just to read the context — before any output. On Gemini 2.5 Pro's 2M tier at $7/M input, a full fill is $14 per request.

For comparison, a typical RAG-style retrieval that fetches the 4 most-relevant chunks (each ~1k tokens) costs about $0.012 of input. A factor of 250×.

Latency: the silent killer

Token-cost is annoying. Latency is the thing that actually kills the UX. Time-to-first-token on a 1M-token input is regularly 8-15 seconds across providers, regardless of model speed. Streaming buys you something, but the user is still waiting nearly 10 seconds before any output appears. That breaks the conversational feel a chat UI depends on.

The same query against a 4k-token retrieved context returns first token in 300-600ms. The gap isn't subtle.

Accuracy: long-context models are not actually good at long context

Every long-context model has the "lost in the middle" problem. They're good at recalling things near the start and end of the context, and noticeably worse at the middle. Benchmarks regularly show accuracy drops of 20-40% for facts buried in the 30-70% range of a 1M-token fill.

The Claude long-context system card from 2026-Q1 calls this out explicitly: the model's "effective context" for needle-in-haystack retrieval is closer to 500k than the advertised 1M, and quality degrades smoothly past that. Other providers don't publish equivalent numbers because the results would embarrass them.

When the 1M window is actually the right tool

Three use cases where long context genuinely beats retrieval:

• Whole-codebase reasoning. "Where is the bug that's causing X?" against a 200k-token codebase, where the bug could be anywhere. Retrieval misses cross-file issues; full context catches them. Worth the $0.60 cost.
• Long-form document analysis. A 300-page contract, a court filing, a multi-chapter book. Chunking and retrieving destroys the document's structure. Long context preserves it. Worth the cost if the task is high-stakes.
• Long-running agent sessions. If an agent is genuinely mid-task with 200k tokens of accumulated context, summarizing-and-restarting loses information. Carrying the full context is the right choice — once.

When you should stick with retrieval + smaller context

• Chat over a knowledge base. Retrieval + 4-8 chunks + a tight system prompt is faster, cheaper, and more accurate.
• Customer support. The relevant info is always a tiny fraction of the corpus. Retrieve.
• Q&A over docs. Same.
• Any UX where latency matters. 8-second first-token is unacceptable in chat.

The caching cheat code

If you are sending 1M tokens repeatedly with only small changes, prompt caching changes the math. Anthropic and Google both offer cached input at ~10% of the normal rate after the first request. A 1M-token context that costs $3 the first time costs $0.30 on every cached request afterward. We covered this in more depth in the original guide to reducing Claude API costs— prompt caching is item #3 there, and arguably the single largest cost lever on Anthropic.

That makes long-context viable for "same big document, many questions" workflows — the cached read is cheaper than the equivalent retrieval pipeline once you factor in retrieval infrastructure cost. But the first request is still $3, and the cache only lasts 5 minutes by default (1 hour with extended-cache pricing on Anthropic).

Long context isn't a replacement for retrieval. It's a different tool, useful when retrieval can't see the structure of the underlying material. Most workloads still want retrieval.

The actual decision tree

When deciding whether to fill the context window, ask in order:

• Is the answer always in one small region of the corpus? → Retrieve.
• Does the task need to understand the whole document's structure? → Long context.
• Will I send this same context many times? → Long context + caching.
• Is per-request latency below 2s critical? → Retrieve, no matter what.

Most teams default to long context because the marketing makes it sound free. It isn't. EcoToken's calibration counts cached vs uncached tokens separately precisely because the difference is the difference between a sustainable bill and an unsustainable one.