The Other Side of the Coin

In a recent article, I made my website AI-agent friendly [1], adding llms.txt, Markdown output, and content negotiation to a Hugo site on AWS. That article was about the producer side: how to serve content that agents can consume efficiently.

But I left a question unanswered: what does it look like from the agent’s perspective?

In this article, I’m building two agents. Same task, same website, same model. The only difference: one reads the web the old way, the other uses the infrastructure I just built. The code is written with the Strands Agents SDK [2], an open-source framework from AWS for building AI agents in Python.

The Kofferklausur, Revisited

In September 2024, a colleague asked an audience: “What is RAG?” I answered: Kofferklausur [1].

For non-German speakers: a Kofferklausur is an open-book exam. You bring your textbooks, notes, everything. The exam doesn’t test what you memorized — it tests whether you can find the right information and reason about it under pressure.

That analogy stuck with me. A foundation model is the student. RAG is the suitcase full of books. The model doesn’t need to memorize every fact — it needs to know how to find the right one and reason about it. Special-purpose tools beat the Swiss Army knife.

The Invisible Error

To test this, I designed five calculations that anyone in business might ask an AI assistant — the kind of questions you’d type into ChatGPT or Claude expecting a quick, reliable answer:

1. Simple arithmetic — 7 × 8 (baseline sanity check)
2. A discount calculation — “What’s the final price of a €249.99 item with 15% off?” (retail, e-commerce)
3. Compound interest — “How much is €10,000 worth after 7 years at 3.5%?” (investment planning)
4. A mortgage payment — “What’s the monthly payment on a €250,000 loan at 3.8% over 25 years?” (the kind of number people make life decisions on)
5. Standard deviation — of a 10-number dataset (basic statistics, common in reporting)

I ran each calculation through two models on Amazon Bedrock: Amazon Nova Micro ($0.046/1M input tokens) and Claude Sonnet 4 ($3.00/1M input — roughly 65x more expensive). Prices are on-demand rates at the time of writing [4]. The choice of models isn’t a judgment on either — both are excellent at what they’re designed for. The point is to show that this is a structural limitation of how language models work, not a quality issue with any specific model. A small model gets it wrong more often. A large, expensive model gets it wrong less often. But neither is computing — both are predicting. The error shrinks with scale but doesn’t disappear, because the architecture is fundamentally probabilistic.