AI Crawlers Are Getting Locked Out of the Web

AI crawlers knocked on the doors of the open web millions of times this summer, and more than one in eight of those knocks went unanswered. Sites on Cloudflare's network denied thirteen point four seven percent of AI bot requests in the third quarter of twenty twenty-six, up from five point nine eight percent a year earlier, according to Cloudflare Radar data analyzed by TechnologyChecker.io and cited by Zen Media in a release dated October 9, twenty twenty-six. The open web is starting to charge AI companies admission — and the toll booths are going up fast.

That denial rate more than doubled in a single year. It means the websites you read, shop on, and learn from are increasingly refusing to let AI systems read them back. If you have ever wondered why your favorite chatbot seems oddly clueless about a recent article or a small creator's work, this is a big part of the answer: the bots were told to stay out before they ever reached the page.

AI crawlers are being blocked at record rates

The scale of the shutout is hard to overstate. Cloudflare CEO Matthew Prince told Wired that the company has blocked more than four hundred and sixteen billion AI bot requests since the start of July twenty twenty-five, as reported by Computerworld — a figure that counts only traffic on Cloudflare's own network, which fronts a sizable share of the internet.

The blockade against AI crawlers is not evenly spread. In a September twenty twenty-six snapshot of more than four thousand robots.txt files, TechnologyChecker.io found that sites blocked GPTBot about one and a half times for every time they allowed it, according to Zen Media's release. OpenAI's search crawler, OAI-SearchBot, was blocked nearly as often as it was allowed. Meanwhile, the share of AI crawl activity that comes from search and user-triggered requests — a person asking a question that sends a bot to fetch a page — rose from about six and a half percent in the second quarter to nearly twelve percent in the third.

Then came the rule change with teeth. In mid-September twenty twenty-six, Cloudflare began blocking AI training crawlers by default for new domains, new zones, and free-tier customers, and also started blocking AI agents on pages that display ads, according to an analysis by RankCLI on Dev.to. The catch, the analysis explains, is that mixed-purpose crawlers — the ones that crawl for search and for AI training at the same time, including Googlebot, Bingbot, and Applebot — get judged by their most restrictive behavior. Block training, and you block the whole crawler. Cloudflare's own network data puts Googlebot's crawl share at about twenty-seven and a half percent in the second quarter of twenty twenty-six, down from more than fifty-seven percent a year earlier.

Why the web is saying no to AI crawlers

The economics are simple: AI companies built hundred-billion-dollar products on content they never paid for, and publishers finally have a lever. Under the old system, blocking AI crawlers was optional. More than a million Cloudflare customers had already chosen to restrict AI bots under that optional setup, according to Infosecurity Magazine. Now the default has flipped — AI vendors must explicitly seek permission and say whether they want the content for training, inference, or search, rather than scraping first and asking later.

Prince has been blunt about what is at stake. In the Wired interview cited by Computerworld, he described AI as a platform shift that will change the entire internet business model, and argued the industry needs fairer rules where AI companies pay for licensed content instead of taking it for free. His sharpest criticism landed on Google, which he says merges its search and AI crawlers so that blocking AI training also erases a site from search results — a practice he called an abuse of its dominant position, as reported by Computerworld.

Zen Media, which sells AI visibility audits, adds a practical warning that reveals how tangled this has become: a firewall, CDN, or bot-protection rule can return a denial even when a site's robots.txt file says the crawler is welcome. The firm's release describes a hypothetical retailer whose content team rewrote two hundred product pages to rank better in AI answers — while AI crawlers never reached the pages at all, because a security rule blocked them months earlier. The fix, in that telling, is a five-minute settings change owned by a different department, not a rewrite.

What the crackdown means for you

If you publish anything online — a blog, a shop, a portfolio, a fan site — this fight is about your work. On one side, blocking AI crawlers protects creators from having their labor absorbed into models that may never credit or pay them. On the other, an AI assistant that cannot see your site cannot recommend it, either. For small creators and independent publishers, invisibility to AI can mean invisibility to the next generation of search. Before commissioning any AI visibility audit, check your server logs and your CDN settings first, as Zen Media advises: access comes before optimization.

There is a counterpoint worth sitting with. The people who rely most on free AI answers — students cramming on no budget, job seekers polishing resumes, anyone without a subscription to anything — are the ones who lose when the knowledge web shrinks. A web where every site charges AI systems rent could become a web where the best answers live behind the tallest fences. The fight for creator compensation is legitimate, and so is the fear of what happens to open knowledge when everything gets a price tag.

What is really happening is a renegotiation of the internet's oldest deal. For thirty years, the bargain was simple: sites let crawlers in, and crawlers sent readers back. AI broke that loop by reading everything and sending far fewer readers anywhere. Now the web is writing new terms — in machine-readable form, enforced at the network's edge, where no polite request in a robots.txt file can override them. The question for the next few years is not whether AI crawlers can read the web. It is what the web decides to charge for the privilege.

More Deep Dives | Related: Anthropic Says Stop Being Cruel to Claude

Sources: The Agile Brand Guide, citing Zen Media and TechnologyChecker.io, RankCLI on Dev.to, Infosecurity Magazine, and Computerworld.