Here is a number worth sitting with: automated traffic now accounts for more than 53% of all web traffic, and bad bots alone make up 40% of it — up three percentage points year on year, driven mostly by AI.
Most of the visitors to your website are not people. That has been quietly true for a while. What changed this year is that the industry started sorting those non-people into categories, and on 15 September some of them begin getting blocked by default.
Worth understanding before it happens to you rather than after.
What actually changes on 15 September
Cloudflare is splitting AI bots into three categories — by what they do, not by who runs them — and changing the defaults:
| Category | What it is | Default from 15 Sept |
|---|---|---|
| Search | Crawls and indexes your content to answer questions about it later | Allowed |
| Training | Collects content to train a model | Blocked on ad-carrying pages |
| Agent | Real-time activity on a specific person’s behalf | Blocked on ad-carrying pages |
The reasoning is that an ad signals a human was meant to land there. As Cloudflare puts it, an ad is a signal that a website owner intended a person to see the page, so on those pages it keeps away the bots that reduce human attention.
Now the part most coverage is skipping. This applies to pages that host ads. If your site is a service business that does not run advertising on its own pages — which describes almost every client we work with — the ad-page rule does not touch you.
What does apply more broadly: the new defaults hit new Cloudflare customers, new sites added by existing customers, and all existing free-plan customers. Paid customers keep the configuration they already have. A lot of small businesses are on Cloudflare’s free plan without ever having thought about it — often because a developer set it up years ago and left.
So: probably not urgent for you. Worth ten minutes to confirm rather than assume.
The category that matters, and why blocking it is a mistake
Search and Training are the ones everyone argues about. Agent is the one that will actually cost you money.
An Agent bot is the thing that loads your pricing page thirty seconds after someone asked an assistant about you. It is not scraping you and it is not training on you. It is a real person’s proxy, sent by a real person, right now. Blocking it is blocking a customer who happened to ask a question in a chat window instead of a search box.
These agents — OpenAI’s Operator, Amazon’s Buy For Me, Perplexity’s shopper — run actual browsers. They render JavaScript, read layouts and fill in forms the way a person would. Which means two things: they are hard to detect by the old signals, and a site that a person can use is broadly a site an agent can use.
The platforms are pushing back, so this is not settled
It would be dishonest to present agentic commerce as a done deal. It is being actively resisted:
eBay prohibits third-party AI agents from autonomously placing orders, from 20 February 2026. Bot-detection vendors now sell purpose-built agent detection that fingerprints LLM platform IP ranges, the timing signature of a reasoning loop, and the anomalies a real browser shows when something is driving it programmatically.
There is a genuine conflict here and nobody has won it. Marketplaces do not want agents transacting because it breaks their economics. Publishers do not want agents reading without paying — Cloudflare is launching a marketplace that pays publishers when their content is actually used in AI answers.
Anyone telling you confidently how this resolves in 2027 is guessing.
What to actually do this month
Short, and none of it is a project:
1. Find out whether you are on Cloudflare at all. Many small businesses are and do not know. If you are on the free plan, the new defaults will apply to you.
2. Decide about Agent traffic deliberately. If someone’s assistant loads your services page on their behalf, do you want that to work? For a service business the answer is almost always yes. Make it a decision rather than a default set by someone else.
3. Check your robots.txt for blanket blocks. A lot of sites picked up
“block all AI bots” rules during the 2024–25 panic. That was aimed at training
crawlers, but the rules were usually written broadly enough to catch the agent
carrying your next enquiry.
4. Look at what your analytics is counting. If more than half of web traffic is automated, some of your “visitors” are not people, and any conclusion you draw from raw session counts is being distorted by that.
Being readable by an agent is not a new discipline
Here is the reassuring part, and it is the same conclusion we reached about generative engine optimisation: there is no new specialism to buy.
An agent reads a page much the way a screen reader does. What helps it is what helps everyone:
- Real HTML. Semantic elements, headings in order, buttons that are buttons.
- Content that exists without JavaScript running. If your prices only appear after three scripts finish, you are gambling on the agent waiting.
- Forms that are actually forms, with proper labels and input types.
- Structured data using the specific type for your business, with service areas named.
- Speed, because agents time out like people do.
Every one of those is on the accessibility checklist and has been for a decade. The agent is just the newest visitor that benefits from work you should have done for the humans.
The honest summary
Do not rebuild your website for AI agents. The traffic does not justify it yet, and the rules are actively being fought over.
But do not accidentally block them either — which is the more likely mistake, and the one that happens by default to a free Cloudflare account on 15 September. Blocking a training crawler is a business decision. Blocking the agent that a customer sent to read your pricing page is just losing an enquiry without ever knowing it happened.
The through-line for the last two years is the same: the sites that do well are the ones that are genuinely fast, genuinely readable and genuinely honest. That was true for humans, it turned out to be true for search engines, and it is turning out to be true for agents. Nobody has yet found a shortcut that beats building the thing properly.
If you would like someone to check the ten-minute list above on your site, we do that free and we will send you what we find whether or not you hire us. If it turns out you are fine, that is what we will tell you — send us the address.
Sources: Cloudflare’s new crawler policy · what the new default does · eBay’s agent ban at checkout · bot traffic share