For two decades, publishers learned to make a trade with search engines: allow crawling, receive visibility and hope some of that visibility returns as traffic. AI systems complicate the bargain because machines can consume content and answer users without sending the same kind of click. Cloudflare is testing a blunt response: put a price on access.1
Cloudflare’s Pay Per Crawl is a closed-beta system that lets participating publishers charge AI crawlers for successful access, with a documented minimum price of $0.001 per retrieval. Cloudflare is also testing Pay Per Use approaches tied to content value in AI search. These are experiments, not evidence that paid crawling is a mature or widely adopted publisher revenue model.1, 2, 3
Cloudflare is testing paid machine access
The minimum is tiny by itself: one tenth of a cent per successful retrieval. Scale is what could make the model meaningful. One million paid retrievals at that minimum would equal $1,000 before any fees or other terms. That is arithmetic, not evidence that a publisher is receiving one million paid crawls.2
Pay Per Crawl changes the access decision
Cloudflare describes Pay Per Crawl as an option for site owners to allow a crawler, block it or require payment. A participating AI crawler that accepts the price can retrieve the content. The system therefore turns crawler access from a binary technical rule into a potential commercial transaction.1, 2
| Model | Crawler access | Publisher outcome |
|---|---|---|
| Free access | Crawler can retrieve allowed pages. | Potential discovery or model/search value, but no per-crawl payment. |
| Block | Crawler is denied under the site’s control rules. | Content access is restricted for that crawler. |
| Pay Per Crawl | Crawler pays an accepted price for successful retrieval. | Publisher receives compensation tied to access under the beta model. |
Access and value are not the same thing
Charging per retrieval is easy to understand but imperfect economically. A crawler might fetch a page that is never used in an answer, or one retrieval might support many valuable interactions. That is why Cloudflare’s newer Pay Per Use experiments explore compensation tied more closely to content creating value in AI search.3
The two approaches answer different questions. Pay Per Crawl asks what access should cost. Pay Per Use asks how value created after access might be shared. Neither has yet become a universal web standard, and the company describes the work as experimentation.
This is the other side of the AI-citation problem
Our AI citation measurement guide separates being cited from receiving a visit. Paid access pushes the question one step earlier: should a publisher be compensated even when the machine consumes content and the user never visits at all?
That debate matters most for publishers whose economics historically depended on referral traffic, subscriptions or ads sold to human readers. If AI answers absorb some discovery activity, content owners may look for a different exchange than “crawl now, maybe send a click later.”
Pricing creates its own coordination problem
A publisher can set a price, but a crawler still has to support the payment mechanism and decide the content is worth buying. If every site uses a different system, transaction costs could become a problem. Cloudflare’s value is partly that it can sit between many sites and crawlers with one control layer.
The market also needs to decide what deserves payment. News, reference material, product pages and personal blogs create different kinds of value. A single per-crawl price cannot automatically express the value of every page or every downstream use.
What would show paid crawling is becoming a real market
- Adoption: a meaningful number of both publishers and AI crawlers participate.
- Repeat transactions: paid access continues after initial tests rather than remaining a demonstration.
- Pricing: publishers can charge enough to matter without making useful crawling uneconomic.
- Attribution: systems can explain what access or downstream use produced the payment.
Cloudflare’s experiment is interesting because it turns a philosophical argument about “AI using the web” into an invoice. Whether that invoice becomes a durable business model is still open. But the web now has a concrete mechanism for testing the idea.
Sources and methodology
Sources checked September 22, 2026. Dates and periods for individual figures are stated beside them.
- Cloudflare: What is Pay Per Crawl? ↗Accessed 2026-09-22
- Cloudflare: Set a Pay Per Crawl price ↗Accessed 2026-09-22
- Cloudflare: Pay Per Use and agentic internet announcement ↗Accessed 2026-09-22
Scope and assumptions
Pay Per Crawl remains a closed beta and Pay Per Use is experimental; no widespread adoption or publisher-revenue outcome is claimed.
The $1,000 example is illustrative arithmetic at the documented minimum price, not observed publisher earnings.
AI-assisted research and editing. Our editorial standards.
Continue reading
4 Numbers to Keep Separate: AI Citations Are Not Website Visits →