Why AI Tokens Are So Expensive — and What Actually Makes Them Cheaper

During some time off in Austria, three of us sat at a marble café table and lost most of an afternoon to a single question: why does an AI token cost what it costs? Two of us have worked deeply on these systems, and the Melange kept coming while we argued over napkins about memory bandwidth, KV caches, and what the machine is doing when it “prefills” a prompt. I walked out with the same feeling I get from the best hallway conversations: the answer is not exotic. It comes down to two things every computer science graduate already carries around. Memory bandwidth, and Big-O.

The spark was a Computerphile episode on why tokens are expensive [1]. It is a good problem statement, and I will not retell it here. I want to go one level further: separate the physical work from the billing arithmetic, then map each mitigation to the cost it actually attacks.

Two phases with different bottlenecks

Autoregressive LLM text generation has two main phases, and they use an accelerator differently.

The first is prefill: the model processes the input prompt and builds the initial key-value cache. The token positions can be processed in parallel, even though causal attention prevents a position from using future content. That turns the work into large matrix operations with substantial arithmetic parallelism. For a sufficiently long prompt or a well-batched workload, prefill is usually compute-intensive and determines time to first token [11] [13] [15]. A serving runtime may still split a long prefill into chunks for scheduling or memory reasons; “parallel” describes the available dependency structure, not a guarantee that every token is physically processed in one indivisible operation.

The second is decode: the model produces the output one token at a time. Each token depends on the tokens before it, so generation cannot expose the same sequence-level parallelism. At each step, the accelerator reads the model weights layer by layer and reads the growing KV cache. Especially at small batch sizes, the arithmetic per byte moved is low enough that memory bandwidth, rather than peak FLOPs, commonly sets the pace [13] [15].

This physical asymmetry helps explain why many hosted models price output tokens above input tokens, although API prices also reflect utilization, capacity, and commercial decisions. It also explains why adding more theoretical compute does not guarantee a proportional generation-speed increase: the bottleneck may be data movement.

The KV cache trades computation for memory

Without a cache, each decode step would have to recompute keys and values for all earlier tokens. Instead, the model stores those projections and reuses them. That store is the KV cache. It removes a large amount of repeated computation, but each new token still has to read and attend over the cached keys and values.

The cache normally resides in the same high-bandwidth accelerator memory used by the weights and active requests. Its size grows linearly with sequence length when architecture and precision are fixed, but there is no universal “bytes per token” number for a 70-billion-parameter model. The relevant factors are the number of layers, KV heads, head dimension, and bytes per cached element [2]:

KV bytes per token ≈ 2 × layers × KV heads × head dimension × bytes per element

The leading two accounts for keys and values. Grouped-query and multi-query attention reduce the number of KV heads, and lower-precision cache formats reduce the final factor. Even so, a long context can consume tens of gigabytes per request [2].

That matters because batching is one of the main efficiency levers in inference. Reading the weights serves every request in the active batch, so a larger batch amortizes that traffic across more generated tokens. A larger per-request KV cache leaves room for fewer concurrent requests. Long contexts therefore carry two costs: more attention work and less space for the batching that improves throughput. PagedAttention addresses the memory-management side by reducing fragmentation and enabling flexible KV-cache sharing [2].

The cheapest token is often the one you do not send. Retrieval can place only the relevant passages in the prompt instead of an entire corpus, reducing both input processing and KV-cache pressure. I explored that lever separately when moving vector search onto Bedrock [9].

The complexity lens: count the right thing

Big-O helps, but only after we say what is being counted.

For standard dense self-attention over an input of n tokens, the attention score and weighted-value operations take Θ(n²d) arithmetic for head dimension d; viewed only as a function of sequence length, that is quadratic [3] [11]. KV-cache memory is linear in n for a fixed model architecture and precision. If a prompt has P tokens and the model generates G more, cached decode attention performs work on the order of Θ(PG + G²): every new token attends over the prompt and the growing output. Other transformer operations add substantial work too, so this is an attention model, not a complete latency equation.

A multi-turn chat introduces a second calculation. Suppose each turn adds roughly c tokens. At turn t, the application sends approximately ct input tokens if it includes the full history. Across T turns, the input-token volume is:

c + 2c + 3c + ... + Tc = Θ(cT²)

That quadratic quantity is relevant to token-metered billing. It is not the same as dense-attention arithmetic. If every turn performs a fresh dense prefill over the whole transcript, the attention portion is instead:

c² + (2c)² + (3c)² + ... + (Tc)² = Θ(c²T³)

This distinction matters. “Tokens submitted,” “floating-point work,” “latency,” and “money charged” are related, but they are not interchangeable complexity measures.

Prefix caching is reuse, not asymptotic magic

The memoization analogy still earns its place, with a boundary. A prefix cache stores reusable internal state for an unchanged prompt prefix, just as memoization stores the result of a repeated sub-computation. On a hit, the serving system avoids recomputing that prefix. The newly added tokens must still be processed, and they must still attend over the cached history.

That means caching can remove a great deal of repeated work without generally turning a growing conversation’s total cost from quadratic to linear. For an equal-sized-turn chat, the uncached new-token volume becomes linear in T, but the cached prefix read on each turn still grows with the transcript. On Amazon Bedrock those cache-read tokens are billed at a reduced, nonzero rate [6]. A discount changes the coefficient; it does not, by itself, change the asymptotic class.

There is another useful case: a fixed system prompt or document reused across many independent queries. Both the uncached and cached bills scale linearly with the number of calls because the prefix length is fixed. Caching can still produce a large practical win by replacing repeated full-price processing with cheaper cache reads and lower time to first token. The right claim is not “caching makes the bill linear.” It is “caching stops you paying the full recomputation price for work the system can reuse.”

The mitigations, grouped by the cost they attack

With the ledgers separate, each optimisation maps to a clearer target.

Decode throughput and weight traffic

• Continuous batching admits new requests as others finish, keeping more useful work in flight; PagedAttention helps make the required dynamic memory management practical [2].
• Speculative decoding uses a faster draft mechanism to propose several tokens that the target model can verify in parallel, reducing the number of expensive sequential target-model steps when acceptance is high [12].
• Weight quantization stores fewer bits per weight, reducing memory capacity and bandwidth requirements, with a model-quality trade-off that must be measured [12].
• Grouped-query and multi-query attention reduce the number of key/value heads, shrinking KV-cache traffic and capacity requirements; inference-scaling experiments show how that lower memory requirement enables much longer contexts [15].

KV-cache memory

• PagedAttention manages the cache in blocks, reducing fragmentation and enabling sharing within and across requests [2].
• KV-cache quantization stores keys and values at lower precision, trading small numerical differences for lower memory use and potentially higher concurrency.
• Sliding-window attention caps how far back a token attends, bounding cache growth for layers that use the window, at the cost of restricting direct access to older tokens.

Prefill scheduling and computation

• FlashAttention tiles exact attention around the memory hierarchy, reducing reads and writes between HBM and on-chip SRAM without changing dense attention’s quadratic arithmetic [3].
• Chunked prefill splits long inputs into smaller scheduling units that can be interleaved with decode, reducing head-of-line blocking.
• Disaggregated prefill/decode takes a different approach: it places the compute-intensive prefill and memory-bandwidth-intensive decode phases on separate worker pools and transfers the KV cache between them [13].
• Cross-request prompt caching reuses matching prefix state. This is the managed lever at the centre of the rest of this post.

Amazon Bedrock prompt caching: reuse you can turn on

Amazon Bedrock prompt caching supports long, repeated contexts across model calls. For models with explicit caching, you place a cache checkpoint after stable prompt content. Other supported models provide automatic or model-specific cache management. The first matching request creates reusable internal state; later requests can read that state and skip recomputing the matching prefix [4] [5] [6].

AWS reports up to 90% lower input-token cost and up to 85% lower latency for supported models and workloads [4] [5]. Skipping prefix recomputation primarily improves the input stage and time to first token. It does not reduce the work or price of the output tokens generated afterward.

The practical mechanics are model-specific, but a few rules recur:

• Put stable content first: tool definitions, system instructions, reference documents, and fixed examples. Put changing user input last. Changes in an earlier section can invalidate cache entries for later sections.
• Cache reads are billed at a reduced rate. Depending on the model, cache writes may cost more than uncached input tokens; ordinary uncached tokens use the standard input rate [6].
• In the Converse API, cacheReadInputTokens and cacheWriteInputTokens expose cache use. Other APIs and model families use different field names, so monitor the usage structure for the API you call [6].
• TTL, minimum prefix length, checkpoint limits, and whether caching is explicit or automatic vary by model. Many models support a five-minute TTL that resets on a hit, and some support a one-hour option. Prompt caching is supported for on-demand inference, not the batch inference API [6].
• The supported-model list evolves. Check the current model card and Bedrock user guide rather than copying a static list into an architecture decision [6].

Good fits have a large stable prefix reused often: chat over a document, coding assistants that repeatedly send the same files, agent workflows with substantial tool definitions, and few-shot prompts with fixed examples [5]. The cache-write premium and TTL mean reuse frequency matters; a prefix written once and never read again can cost more than an uncached call.

The honest boundary is simple: prompt caching reduces repeated input processing and input-token rates. It does not make cached reads free, and it does not make decode cheaper. Workloads with long repeated prompts and short outputs benefit most; workloads dominated by long generation still pay for that generation.

The silicon half

Hardware provides another lever. AWS Inferentia is purpose-built for inference, while Trainium is designed primarily for training and also supports large-scale inference [7] [8]. A Trn2 instance, for example, combines 16 Trainium2 chips with 1.5 TB of HBM3 and 46 TB/s of aggregate memory bandwidth [14]. Those are relevant resources for models whose performance depends on keeping large weights and caches close to the compute units.

The hardware choice does not replace the software techniques above. Quantization changes how many bytes must move. Batching changes how many requests share a weight read. Cache management changes how much concurrent state fits. The accelerator determines how quickly the remaining work can run. These levers stack, but their value has to be benchmarked against the model, context lengths, output lengths, and latency target. Pope et al. analyze these interactions directly through partitioning, batch size, quantization, and attention architecture on large-model inference [15].

If You’re Running This on AWS

For an Amazon Bedrock workload, start with the request shape. Put stable content before dynamic content, enable the caching mode supported by the selected model, and measure cache writes, cache reads, uncached input tokens, output tokens, time to first token, and total latency [6]. Compare the write premium against subsequent discounted reads instead of assuming every cache entry saves money.

If you host a model yourself on Amazon SageMaker AI or Amazon EC2, the lower-level levers become yours: continuous batching, quantization, PagedAttention, chunked or disaggregated serving, and hardware selection. Inferentia and Trainium are options in that design space; they are not knobs that you attach to a managed Bedrock model invocation.

The recurring pattern is still reuse, but the accounting has to stay honest: reuse weights across a batch, reuse keys and values across decode steps, reuse prompt-prefix state across requests, and then measure the ledger each technique actually changes.

A thread worth pulling

What made that afternoon in Austria stick was how much of modern AI cost comes back to the classic toolkit: memory hierarchies, asymptotic analysis, batching, and caching. The useful move is not merely spotting an O(n²) term. It is naming the variable, separating computation from billing, and finding the repeated work the system can safely reuse.

I made a neighbouring argument before, that compression and intelligence are two views of the same thing [10]. If this framing is useful, this may not be the last post in that vein.

Where does the cost model bite hardest in your workloads: repeated input processing, KV-cache capacity, or sequential decode?

Sources

• [1] Computerphile, “Why AI Tokens Are So Expensive” — video published 2 Jul 2026.
• [2] Kwon et al., “Efficient Memory Management for Large Language Model Serving with PagedAttention” — SOSP 2023.
• [3] Dao et al., “FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness” — 2022.
• [4] Amazon Web Services, “Amazon Bedrock prompt caching” — product overview.
• [5] Amazon Web Services, “Effectively use prompt caching on Amazon Bedrock” — 7 Apr 2025.
• [6] Amazon Bedrock User Guide, “Prompt caching for faster model inference” — current mechanics, APIs, TTL, usage, and billing behavior.
• [7] Amazon Web Services, “AWS Trainium” — accelerator overview.
• [8] Amazon Web Services, “AWS Inferentia” — inference accelerator overview.
• [9] Stefan Christoph, “Vector Search, From the Whiteboard to the Cloud” — retrieval and context selection.
• [10] Stefan Christoph, “Compression Is Intelligence” — related argument on representation and intelligence.
• [11] Vaswani et al., “Attention Is All You Need” — transformer architecture and attention complexity.
• [12] AWS Prescriptive Guidance, “Model optimization techniques” — speculative decoding and quantization.
• [13] Amazon Web Services, “Disaggregated prefill and decode for LLM inference on SageMaker HyperPod” — phase characteristics and separate worker pools.
• [14] Amazon EC2 Trn2 instances — 16 Trainium2 chips, 1.5 TB HBM3, and 46 TB/s aggregate memory bandwidth.
• [15] Pope et al., “Efficiently Scaling Transformer Inference” — analytical and empirical study of partitioning, latency, utilization, quantization, batching, and multi-query attention for large generative Transformer inference.

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AI
• LLMs
• AmazonBedrock
• InferenceCosts
• KVCache