The crawl deal is broken
Last week Elon Musk quote-tweeted a Cloudflare forecast with "AI agentic Internet traffic will obviously VASTLY exceed human usage. Not a close call at all. Cloudflare's forecast is accurate." The headline everyone ran with was that bot traffic will be a thousand times human traffic within five years, and that humans are about to become "a rounding error on the internet".
Both quotes are real. Both are missing the half that matters.
Here is what Cloudflare CFO Thomas Seifert actually said on the earnings call, as reported by The Register:
To give you a sense of how this trend is playing out, and with the big caveat that I have called it wrong at every point along the way, if the current trends continue, we think in five years, non-human traffic will be as much as 1,000 times as much as human traffic.
"I have called it wrong at every point along the way." That clause did not survive contact with the timeline. Musk called a forecast accurate that its own author prefixes with an admission of repeated error.
And the rounding error line comes with its own correction, from the same man:
In other words, humans will be a rounding error. Not because human traffic goes down, but that's just how fast we're seeing non-human traffic grow.
Human traffic is not falling. The denominator is exploding. Those are very different stories, and only one of them is a crisis.
The verifiable parts hold up and are interesting enough without the drama. Machine traffic passed human traffic on Cloudflare's network in May 2026, a year earlier than Cloudflare's own projection of 2027. Non-human requests were 57.5% of volume in the second quarter.
I build SEO tools for a living, so people have been asking me what this means all week. My answer is that 57.5% is not the number to look at.
The number that matters
In June 2025, Cloudflare started publishing a metric called the crawl-to-refer ratio. It is exactly what it sounds like: how many pages a platform crawls for every visitor it sends back. Requests for your HTML on one side, requests carrying that platform's hostname in the Referer header on the other.
It is the only metric that describes the actual bargain of the web, and the bargain is simple. You let me read everything you have written. I send you readers. Everything built on top of the open web for twenty-five years, including the entire industry I work in, rests on that trade.
Cloudflare Radar, for the window ending 21 July 2026:
| Platform | Pages crawled per referral |
|---|---|
| ~4.6 : 1 | |
| Microsoft | ~35 : 1 |
| OpenAI (GPTBot) | ~217 : 1 |
| Perplexity | ~225 : 1 |
| Anthropic (ClaudeBot) | ~2,237 : 1 |
Google reads about five of your pages and sends you a person. That is the deal as it was written. Anthropic reads two thousand two hundred.
That is the story. Not the volume of machines, the exchange rate. The crawling continues exactly as before. The counterparty has quietly stopped paying.
What I got wrong in May 2025
Fifteen months ago I published a guide on optimising content for AI search. I reread it this week.
The mechanics hold up fine. Structure your content so a model can extract it, answer the question in the first hundred words, use proper schema, build topic authority. None of that has aged badly, and I would write most of it again.
What aged badly is the assumption underneath it. The whole guide treats being cited as the win condition. I even listed the metrics to track, and one of them was "Source Traffic: visits from AI-cited content", as though citation and traffic were two ends of the same pipe.
They are not the same pipe. They are barely the same building. You can be the cited source in an answer that reaches a million people and receive a handful of visitors, because the answer was the product and your page was the raw material. I wrote a guide about how to be quoted, at a moment when being quoted was in the middle of decoupling from being read.
I also wrote that 30% of traditional organic search real estate would disappear by the end of 2025. That framing was too tidy. The real estate did not disappear, it stopped converting, which is worse and harder to see in a dashboard.
None of that makes the guide useless. It makes it a guide to visibility written by someone who still assumed visibility paid.
Three things the thread missed
Three things got dropped on the way to the timeline, and all three cut against the panic.
The ratios are improving, and fast. ClaudeBot sat around 70,900 pages per referral when Cloudflare first published the metric in June 2025. It was roughly 57,000 in January 2026. It is in the low thousands now. OpenAI went from four figures to the mid hundreds over the same period. The direction of travel over thirteen months is a two-order-of-magnitude improvement, because these platforms started shipping surfaces that actually link out. That is the opposite of the trend everyone is describing.
The measurement overstates the problem. This is Cloudflare's own caveat on its own metric, and almost nobody quotes it:
traffic referred by Claude's native app does not include a
Referer:header
Referrals from native apps are invisible to this calculation. Cloudflare says plainly that the ratios "may overstate the respective ratios, but it is unclear by how much". So the real exchange rate is better than the table above, by an amount nobody can currently measure.
The messenger sells the remedy. The 1,000x forecast was made on an earnings call by a company that sells bot management and controls for AI crawlers. That does not make it false. Seifert's numbers are the best public data we have and I have just built half this post on them. But a projection of a machine flood, delivered by the firm selling the flood barriers, deserves to be read with that context attached. Nobody who reshared it last week attached it.
What actually changed
One thing changed, and it is structural rather than technological.
For twenty-five years, crawling and referring were bundled. One actor did both, the ratio was around five to one, and nobody had to think about it because the incentives lined up on their own. Search engines needed your content to have a product, and you needed their traffic to have a business.
Those two functions have come apart. Reading your site and sending you readers are now separate activities, performed by different companies, with wildly different economics. Some of them send almost nothing back, and there is currently no mechanism that obliges them to.
That is what "the crawl deal is broken" means. Not that machines outnumber us, they were always going to, and a crawler was never a reader. It means the compensation loop that made publishing on an open web rational has been unbundled, and it is being reassembled in public, badly, one earnings call at a time.
What I tell people now
None of this is new advice. It just has a number attached to it now.
Measure your own ratio. Every argument in this post is about someone else's network. Your logs are right there. Group requests by user agent, count GPTBot, ClaudeBot, PerplexityBot, Bytespider, Amazonbot and Googlebot against the sessions each one actually sends you, and you will know your exchange rate rather than the internet's. If you sell anything online, that number belongs on the same dashboard as your conversion rate. I am going to publish mine.
Stop optimising for citation as if it were traffic. Citation is brand distribution now, not acquisition. It is worth having, it builds the branded searches and the direct visits that do convert, and it should be measured as an awareness channel with a long lag. Budget for it accordingly, and stop reporting it as if it were a click.
Own the destination. The assets that survive an unbundling are the ones a summary cannot replace. A newsletter list, a product, a community, a tool people open rather than read about. Anything whose entire value is a paragraph a model can restate is competing directly with the restatement, and losing.
None of this is a reason to write less. If the exchange rate on an average page is heading toward zero, the pages worth making are the ones that are not average, and that has been true since long before GPTBot existed.
But stop calling the forecast accurate. The man who made it already told you it is not.