Why Doesn't the Web Just Use HTTP? It Does — and It Doesn't

The question that keeps coming up

Every time I show someone the managed paywall at the edge, the same question lands within a minute: if HTTP already serves the article, and x402 can now charge for it, why is the industry building separate “LLM content ingest APIs”? Why not just use HTTP?

It is a good question. It is also framed one layer off.

This is Part 4 of a series. Part 1 covered how HTTP 402 sat unused for thirty years until agents gave it a payer [6]. Part 2 built the agent that pays [7]. Part 3 rebuilt the publisher on AWS WAF’s managed monetization so the same agent paid with no shim [8]. Those posts settled the how. This one answers the why not just HTTP, and the honest answer is that the web mostly is using HTTP. Where it isn’t, the reason is business structure, not a hole in the protocol.

One article, six layers

Follow a single article as an AI agent tries to reach it. It passes through a stack of gates, and each “competing standard” turns out to sit on a different one:

Read it top to bottom. Enforcement decides whether an unknown bot gets in at all. Preferences declare what you’d like done with the content (and, per the IETF, “are not a security mechanism” [1]). Terms say what you’re licensed to do. Payment collects the money. Delivery hands over the bytes.

The “API vs HTTP” debate collapses two of these, terms and delivery, into one word, “API,” and then compares that blob to “HTTP,” which is really layers 4 and 6. Once you separate them, most of the apparent conflict evaporates.

The plot twist: the spec everyone cites isn’t a delivery API

The standard most often held up as the alternative to plain HTTP is the IAB Tech Lab’s CoMP. So I read the 1.0 spec end to end, and the first surprise is in the title. It is not called an ingest API. It is called the “Content Metadata Marketplace Supply Specification.”

CoMP is an object model for describing and negotiating an offer: who is asking, for what intended use, under which license, at what pricing basis, with what provenance. What it explicitly puts out of scope is the punchline [2]:

And the field that answers how do I actually get the bytes (Retrieval.type) enumerates HTML, RSS, API, MCP, NLWeb, XML, NewsML. Plain HTML over HTTP is a first-class delivery option inside the standard [2].

So CoMP does not ask a publisher to abandon HTTP. It standardizes the machine-readable offer sheet that sits in front of whatever delivery already exists. It is a layer-3 terms standard that points at layer-6 delivery and refuses to specify payment. x402 is a layer-4 payment protocol that gates layer-6 and knows nothing about terms. They intersect nowhere, which is why “CoMP vs x402-over-HTTP” is a category error, and why AWS can ship WAF monetization and a CoMP-style guidance without them competing.

Where plain HTTP genuinely wins

Take the pro-HTTP case seriously, because for one whole pattern it is simply right.

For per-request, inference-time access (an answer engine grounding today’s article, a shopping agent checking a price, a RAG pipeline pulling one page), HTTP plus x402 at the edge is complete and shipping. The decisive property is that the payment is the credential: the buyer needs no prior account, no API key, no per-publisher integration [3]. An agent that has never seen your site can pay for one article and move on. Micro-amounts settle in under a second, and the edge verifies before your origin sees the request. AWS WAF’s Monetize action and Cloudflare’s Monetization Gateway both ship exactly this [3] [4].

HTTP also quietly wins the boring parts. ETag/304 re-crawl efficiency beats any bespoke /query endpoint’s hand-rolled delta logic, and content negotiation can serve markdown to agents and HTML to humans from one URL. Nobody has to build a second surface to charge an agent for an article.

Where it genuinely breaks down

And now the honest other half, because a builder who only tells the flattering side isn’t worth reading.

• Bulk training licenses. “The last ten years of the archive, quarterly refreshes, at a negotiated price” is corpus-shaped. Per-request 402 is the wrong economic unit (nobody meters fifty million pages at a cent each with per-request settlement overhead), and crawling the rendered site is the wrong transport. This is contract-and-packaged-delivery territory.
• Usage rights after the 200. HTTP authorizes access; it says nothing about what you may do with the bytes. A 402 payment buys a response, not the distinction between “licensed for grounding with citation” and “licensed for training.” That distinction is the terms layer: RSL’s permitted-use vocabulary [5], CoMP’s function/sub-function declarations. HTTP can transport a license URL (a Link: rel="license" header), but the semantics and the audit trail need a standard above HTTP.
• Usage reporting. Publishers want to know what happened after ingestion: which content fed which answers. Nothing at the transport layer can see a model’s internals. That requires cooperation (CoMP defines a reporturl) or output-side attribution, and it is why high-value relationships can’t be fully anonymous.

Here is the scoresheet I keep coming back to:

The real reason isn’t the protocol

If HTTP plus x402 covers per-request access, why do publishers and their standards bodies keep reaching for marketplaces and negotiated deals? Not for a technical reason, but for a market-power reason, and this is the part I’d argue is the actual answer to the whole question.

The money flowing to publishers today runs through deals, not micropayments: bilateral licensing with large publishers, subscription revenue pools, intermediary rate-card clearing. A CFO budgeting a newsroom wants committed license fees, not a stochastic stream of one-cent events whose volume the buyer fully controls. Atomized per-page pricing gives a single site no bargaining position (a lab can just skip it), so the institutional responses are collective by design: RSL created a nonprofit collective in the mold of a music-licensing society [5], and the IAB openly expects a marketplace to form [2]. A perfect payment protocol does not aggregate supply. That is my read, labeled as analysis, but the vendor behavior lines up with it.

There is a legal dimension too. A negotiated license, or even a machine-issued one with declared intended use, creates evidence of who agreed to what. Terms-first framing (“commercial agreements before crawling” [2]) maximizes that position. An anonymous payment that implies unclear rights arguably weakens a future claim. The copyright card is the publisher’s main asset, and the marketplace path protects it.

The hybrid end-state

Put the layers back together and the end-state isn’t a winner. It’s a stack:

• Edge-enforced 402/x402 for anonymous, inference-time access: the front door on the existing site.
• A machine-readable terms layer (RSL license link and/or CoMP usage declaration) that the 402 response points at, so a paying agent also learns what it’s licensed to do.
• Marketplaces and collectives for aggregation and bulk training deals, where the real negotiated money is.

The pieces already interlock: CoMP’s authentication list includes Web Bot Auth, RSL supports HTTP-header attachment, and the edge products pair payment with identity. In the next post I build the middle layer directly: a small, public publisher whose 402 carries its license terms, so payment and permission arrive in one exchange. The code is on GitHub.

If you’re running this on AWS

The concrete version of the hybrid is two AWS pieces that are halves of one architecture, not competitors. AWS WAF AI traffic monetization is the per-request front door: a Monetize rule action on a CloudFront web ACL returns the 402 with pricing, verifies the signed x402 payment at the edge, and settles through a facilitator [3]. A CoMP/RSL-style terms document served from the same origin is the licensed-partner and audit surface. One publisher stack, two relationship types: the anonymous tail pays per request; the negotiated partners get packaged terms. Frame it as layers, and the “API vs HTTP” choice stops being a choice.

So: does the web just use HTTP? For delivery and per-request payment, yes, and increasingly so. For terms, reporting, and bulk licensing, it uses a thin layer above HTTP and a marketplace beside it. The protocol was never the constraint. Market power was.

What would make you, as a publisher, trust a per-request rail with real revenue? Or is the negotiated deal always going to win? I’d like to hear it.

Sources

• [1] IETF draft-ietf-aipref-vocab-06 — AI usage preferences vocabulary — defines train-ai / search and states preferences are not a security mechanism.
• [2] IAB Tech Lab CoMP 1.0 specification — the “Content Metadata Marketplace Supply Specification”; out-of-scope list (payment, licensing, blocking, discovery) and Retrieval.type including HTML.
• [3] AWS WAF AI traffic monetization — the Monetize action returns HTTP 402 with pricing and verifies the signed payment at the edge.
• [4] Cloudflare Monetization Gateway announcement — x402 charging for pages, datasets, APIs, or MCP tools; x402 governed by the x402 Foundation under the Linux Foundation.
• [5] Really Simple Licensing (RSL) — per-URL license terms and the RSL Collective.
• [6] HTTP 402: The 30-Year Placeholder (Part 1)
• [7] I Built the Agent That Pays (Part 2)
• [8] Getting Paid by Agents (Part 3)

About the Author

Stefan Christoph is a Principal Solutions Architect at AWS, focused on agentic AI, media & entertainment, and helping builders move from demo to production. He writes about AI architecture, developer productivity, and the future of software.

This is a personal blog. Opinions expressed here are my own and do not represent the views or positions of my employer.

Learn more →

• AIAgents
• AgenticWeb
• ContentMonetization
• HTTP402
• X402
• AWSWAF
• Publishing