The 31% Problem: AI Bots Are Reading Your Site and Paying You Nothing

Your best-performing article this year was probably read more times by machines than by people. Not metaphorically. By request count.

The machines did not see an ad, did not subscribe, did not click anything, and did not appear in the part of your analytics you look at. They did consume bandwidth, distort your averages, and take the thing you spent money producing.

This is the newest line item on a publisher's balance sheet, and most publishers have not noticed it is there.

The surge, in numbers

DataDome's 2025 analysis found AI-driven traffic grew more than fourfold in the first eight months of the year — from 2.6% of verified bot traffic in January to over 10% by August. OpenAI's crawlers alone accounted for 1.7 billion requests in August.

Vercel's infrastructure-level data puts GPTBot at roughly 569 million requests per month and Anthropic's Claude crawler at about 370 million.

DoubleVerify recorded general invalid traffic rising 86% in the second half of 2024, attributing the increase substantially to AI crawlers, with AI scrapers representing 16% of known-bot GIVT across that year.

By Q3 2025, AI bots were approximately 31.5% of web traffic.

Whatever your view on whether this is fair, it is now a material share of what hits your infrastructure, and it behaves nothing like the traffic your business was built around.

Why the number is hard to pin down

Different sources give different figures for AI traffic share, and the spread is wide enough that it is worth explaining rather than picking a favourite.

Some measurements count only self-declared AI crawlers. Some count all automation that behaves like a scraper. Some count requests, others sessions, others bandwidth. A site with a large archive and good search visibility attracts far more crawler attention than a small one, so the share also varies enormously by publisher.

Treat any single percentage as directional. What is not in dispute across every source is the gradient — sharply up, over months rather than years — and that the return traffic has not moved with it.

The extraction imbalance

Search crawlers have always taken content without paying. The arrangement was tolerable because it was an exchange: Google indexed your work and sent readers back, and those readers saw ads, subscribed, and returned.

AI crawling breaks that exchange, and the numbers are not close.

Four thousand times. That is the difference between a distribution relationship and an extraction one, expressed as a ratio.

And the direction is structural rather than accidental. A search engine's product is a list of links; sending you traffic is the product working. An assistant's product is an answer; sending you traffic is the product failing to be useful. The referral rate is low because the referral is not the point.

The bandwidth bill nobody itemises

Before the licensing argument, there is a plain operational cost that shows up on an invoice.

Crawlers do not benefit from your caching strategy the way readers do. They request deep archive pages that a human audience has not touched in years and that no CDN edge is holding warm. They ignore the pagination and navigation patterns your infrastructure was sized for. On a large archive this is not a rounding error — it is origin traffic, at scale, for content that generates no revenue on delivery.

Publishers running on managed hosting with metered bandwidth have been quietly paying for this increase for two years, usually attributing it to growth.

Why your analytics are lying to you

This is the part that costs publishers money today, quietly, before any licensing question is settled.

Much of this crawler activity is classified as general invalid traffic. That means it is filtered out of billable impressions — which is correct — but it also means it sits inside your traffic data as a large, non-human population that behaves nothing like your audience.

The consequences compound:

• Your RPM baseline is wrong. Revenue divided by inflated pageviews reads as declining monetisation when nothing about your monetisation changed.
• Your engagement averages are diluted. Crawlers do not scroll, do not return and do not spend time. Blended into your averages, they make a healthy audience look indifferent.
• Your content decisions get distorted. If crawler interest concentrates on certain kinds of article, those articles look like traffic winners in a report that does not separate machines from readers.
• You can trip your own fraud alarms. A large automated population arriving at your tags is exactly the shape enforcement systems look for, and "it was only AI crawlers" is not a defence you get to make in advance.

The polite ones and the rest

There is an important split inside this traffic, and it determines what you can actually do.

A crawler that identifies itself — GPTBot, ClaudeBot, PerplexityBot — is trivially blockable. One line in robots.txt, or a rule at the CDN, and it stops. These are the well-behaved actors, and the fact that they declare themselves is precisely what makes them controllable.

A crawler arriving with a user-agent string describing an ordinary Chrome browser on Windows, from a residential IP address, is not distinguishable from a reader by any declared property. Nothing in the request identifies it.

That is the uncomfortable arithmetic of user-agent filtering, and it is why behavioural detection has become table stakes rather than a refinement. The signals that survive disguise are the behavioural ones — request patterns across a session, breadth of content accessed, the total absence of the small human behaviours that accompany reading.

The one place the traffic is worth something as-is

There is a counter-argument to all of this that deserves a fair hearing, because it is the strongest one publishers make in the other direction.

Being present in an assistant's answers has value even without a click. A reader who asks an assistant about a subject and receives an answer attributed to your publication has encountered your brand, and some meaningful fraction of the time will come looking for you later, directly.

That effect is real. It is also, currently, almost entirely unmeasurable — which is precisely the problem. "It probably helps in ways we cannot see" is exactly what the industry told itself about social platforms for a decade, and the settlement of that argument was not favourable to publishers.

The reasonable position is neither reflexive blocking nor unpriced generosity: treat citation value as real, insist that it be measurable, and decline to accept unmeasurable benefit as payment.

robots.txt is a request, not a lock

Worth being precise about, because a great deal of publisher advice treats these as the same thing.

robots.txt is a voluntary convention from 1994. It has no enforcement mechanism. A crawler reads it and decides whether to comply, and compliance is a policy choice by the operator rather than a technical constraint. Major AI companies have published crawler names and do generally honour disallow rules for those named agents — which is genuinely useful and worth using.

But a file that works only against parties who choose to respect it protects you from exactly the crawlers you were least worried about. It is a sign on a door, not a lock.

The layer that actually enforces anything is the CDN or edge. Blocking, rate-limiting and challenge rules applied before the request reaches your origin do not ask permission. Several providers now ship AI-bot controls as a toggle, and some have introduced pay-per-crawl arrangements — the first serious attempt to convert this traffic into a price rather than a binary allow/deny.

llms.txt, and why a second voluntary file is not the answer

An emerging convention proposes llms.txt: a file describing a site's content in a form convenient for language models, sometimes framed as a way to state licensing terms.

It is a reasonable idea and it may become useful for discovery. It is not a control. It inherits the same property as robots.txt — a voluntary declaration honoured by whoever chooses to honour it — and stating terms in a file that carries no enforcement is closer to posting a price list than to signing a contract.

Useful as a signal of intent, and worth publishing for that reason. Not a substitute for either edge enforcement or an actual agreement.

The agentic wrinkle

A newer complication is worth understanding because it will shape the next two years of this argument.

Neither DoubleVerify nor IAS currently separates agentic visitors — an AI agent browsing on behalf of a specific user — from other general invalid traffic. They are pooled.

That matters because it creates an accountability gap. When a publisher's trap file catches an AI company accessing content in violation of stated rules, the company can respond that the access came from an AI browser user — a human directing an agent — rather than from its own crawling infrastructure.

That defence may often be true. It is also conveniently unfalsifiable from the publisher's side, and it puts a cloud over exactly the question a licensing negotiation needs answered: was this content taken for training, or read on behalf of a person?

The industry response taking shape through 2026 is authenticated crawling — cryptographic verification of crawler identity rather than a self-reported user-agent string. That is the right direction. It replaces a claim anybody can make with a proof only the real party can produce, and it is the only version of this that ends with an enforceable agreement rather than an argument.

Two things are happening at once, and they point opposite ways

The commercial picture is genuinely unsettled, and a publisher should hold both halves of it.

On one side, a growing set of licensing agreements between publishers and AI companies — some substantial, most undisclosed in value — establishes that this content has a price and that large publishers can charge it. That is the direction of travel and it did not exist two years ago.

On the other, litigation over training-data scraping is working its way through courts in several jurisdictions, with the central questions — whether training constitutes fair use, whether output that reproduces source material infringes — still open. Those outcomes will move the negotiating position more than any deal signed in the meantime.

Meanwhile AI answer surfaces in search results have compressed referral traffic for exactly the query types publishers historically won: the factual lookup, the how-to, the definition. That is revenue leaving before any licensing revenue has arrived to replace it.

From cost to line item

Here is the reframe, and it is the reason this article sits under Monetization rather than Ad Fraud.

A third of your traffic is machines consuming your content and returning essentially nothing. Today that is a cost — bandwidth, distorted analytics, uncompensated extraction. The blocking conversation treats it as a loss to be minimised.

But a large, identifiable, commercially motivated population arriving at your content and taking something of value is not obviously a loss. Described in any other context, that is demand.

The question is not only "how do I stop this". It is "why is this the only category of visitor I have never tried to charge".

That three-way decision is the important part of the design, and it is worth being precise about why. Blanket blocking costs you nothing today and possibly costs you a great deal later, when assistants are a real discovery surface and you are absent from them. Blanket allowing is what everybody does now, for free. A per-crawler decision — with search always exempt, because search still sends readers — is the only option that treats the situation as a commercial one.

Why this favours large publishers, and what smaller ones can do

Worth being honest about: the licensing conversation as it currently exists is a conversation between very large publishers and very large AI companies. A regional site with two hundred thousand monthly readers is not going to negotiate a bilateral deal with a frontier lab, and pretending otherwise would be selling a fantasy.

What is available at smaller scale is narrower but real:

• Measurement, which costs nothing and is the precondition for everything else. A publisher who can state what was taken has a position; one who cannot has an opinion.
• Collective arrangements. A network representing many publishers can present an aggregate that individually none of them could. This is the same logic that makes ad networks exist, applied to a new buyer.
• Technical control. Deciding per-crawler is available to anyone with a CDN and does not require a counterparty to agree to anything.

What to do this quarter

Concretely, in order of effort:

1. Segment declared bots out of your reporting baseline. Free, and it fixes every metric you are currently misreading. Do this first regardless of your view on anything else in this article.
2. Measure what is actually arriving. Which crawlers, how often, which sections of the archive. You cannot value an asset you have not counted.
3. Decide your position per crawler, not globally. Search exempt. Assistants that send meaningful referrals treated differently from those that send none. Training crawlers as a separate question again.
4. Write it down somewhere enforceable. robots.txt is a request, not a control — but a stated position you can point to is the precondition for every conversation that follows, including the legal ones.
5. Watch the undeclared traffic behaviourally. This is the hardest and the one that needs tooling, because by definition nothing in the request will tell you.

None of this resolves this year. What a publisher can control is whether, when it does resolve, they are holding a measured record of what was taken or a general sense that something was. That difference costs nothing to secure and cannot be created retroactively — and it is the difference between arriving at a negotiation with a position and arriving with a grievance.