Why AI Tokens Are So Expensive — and What Actually Makes Them Cheaper

Wed, 15 Jul 2026 00:00:00 +0000

TL;DR: Autoregressive LLM generation has two phases with different bottlenecks: prefill processes input positions in parallel and is usually compute-intensive, while decode produces tokens sequentially and is often limited by memory bandwidth. Long contexts also grow the KV cache, consuming memory that could otherwise support more concurrent requests. In a multi-turn chat, repeatedly sending the full history makes the number of input tokens grow quadratically with the number of similarly sized turns; dense-attention arithmetic is a separate measure and grows faster still. Amazon Bedrock prompt caching does not make those costs disappear or change their Big-O by fiat: it reuses matching prefix state and charges cache reads at a reduced rate, with AWS reporting up to 90% lower input-token cost and up to 85% lower latency for supported models and workloads.

During some time off in Austria, three of us sat at a marble café table and lost most of an afternoon to a single question: why does an AI token cost what it costs? Two of us have worked deeply on these systems, and the Melange kept coming while we argued over napkins about memory bandwidth, KV caches, and what the machine is doing when it “prefills” a prompt. I walked out with the same feeling I get from the best hallway conversations: the answer is not exotic. It comes down to two things every computer science graduate already carries around. Memory bandwidth, and Big-O.

schristoph.online

Why AI Tokens Are So Expensive — and What Actually Makes Them Cheaper