Websites Are Blocking AI Agents Across the Web

The pitch for AI agents has always been simple: tell your assistant what you want, and it goes off and does it. Book the flight, buy the shoes, clean out the inbox. But across the web this week, the answer coming back is increasingly a closed door, and websites blocking AI agents are the reason. According to reporting by Android Headlines on The Verge's October 17 investigation, users of Meta's Muse, xAI's Grok, OpenAI's Dots, and the Instinct agent are all running into websites that refuse to let their agents complete tasks. The list of sites blocking AI agents now spans review platforms, airlines, banks, and the world's largest retailer — and it is growing faster than the agent makers can negotiate around it. Websites blocking AI agents has gone from a fringe complaint to the default posture of the commercial web.

The block list is getting longer

The most striking confirmations are the ones companies have put on the record. For websites blocking AI agents, the justifications fall into two buckets: money and security. Yelp told TechCrunch that it only allows AI agents that pay for its data licensing program — access for a price, take it or leave it. Delta Air Lines is preventing agents from booking tickets on behalf of users, though the airline has not detailed the exact restrictions. And in banking, JPMorgan Chase and Charles Schwab have both confirmed they are blocking AI agents from accessing customer accounts, citing security concerns, as covered by zot.news and detailed in this breakdown of the bank blocks.

Amazon remains the heaviest door. The company previously sued AI startup Perplexity over its agentic shopping feature, and reports collected this week describe agents from Meta's Muse, OpenAI's Dots, xAI's Grok, and Instinct all failing on Amazon. One widely shared post showed an Amazon notice reading that continued access by an unauthorized AI agent violates the company's conditions of use. Beyond the confirmed cases, users report losing access that used to work: Cars.com and Amtrak have stopped cooperating with some agents, according to user reports cited by The Verge, though neither company has confirmed a deliberate policy. Reddit threads describe Google flagging accounts that use Meta's Muse for suspicious activity, and users report checkout failures on Walmart despite its partnership with Meta.

The pattern is bigger than any single company's policy. As one analysis of the trend put it, the promised errands — booking dinner reservations, cleaning inboxes, buying shoes — all assume a web that lets the agent in. When websites are blocking AI agents at the door, capability stops mattering.

Access is becoming a business model

Not every site blocking AI agents wants them gone — some simply want to be paid. Yelp's paid data licensing program is the clearest example of the new tollbooth: if an agent wants the reviews, somebody pays. The question of who pays, and whether the cost gets passed to the user, is now part of every agent deployment's unit economics.

Meta has taken the partnership route instead of the fight. The company responded to the blocks in a blog post earlier this month with the argument that turning away a personal agent means turning away the customer behind it, and that it would rather work with businesses on the underlying concerns than watch them lose that customer. Instead of contesting Amazon's restrictions, Meta has signed retail partners — Gap, Sephora, Best Buy, and Walmart — where its Muse agent is explicitly allowed to shop. Shopify, meanwhile, has opened its network of storefronts to AI agents, betting that more checkout traffic is better than the alternative.

The split creates a two-tier web for agents: negotiated corridors where the big assistants are welcome, and everywhere else, where they are treated like scrapers. Websites blocking AI agents over revenue are effectively setting the price of machine access — and only the largest agent makers can afford the negotiation. For smaller agent developers without Meta's deal-making muscle, the corridors may stay out of reach — which is why parallel infrastructure matters. Celestia's Eden chain, built as a sandbox where agents can transact on-chain, and Tether's investment in the AIsa agent-payments platform are both attempts to give agents rails that do not depend on any single website's permission.

Security concerns meet the anti-bot dragnet

The banks' rationale — security — is the hardest to argue with, and the hardest to engineer around. An agent logging into a bank account on a user's behalf looks, to every fraud system ever built, like credential sharing at best and account takeover at worst. JPMorgan Chase and Charles Schwab are not drawing a line against agents specifically so much as enforcing a rule that predates them: nobody but the account holder touches the account.

Much of the blocking AI agents experience, though, is not deliberate policy at all. Chris Canal, chief executive of the AI safety evaluator EquiStamp, told The Verge that plenty of obstacles were built to stop malicious bots, not helpful ones — and agents trip the same wires. Anti-bot systems from providers like Cloudflare, which now blocks AI agents by default on ad-supported pages for new and free-plan sites, routinely flag agent traffic as suspicious. The same CAPTCHAs and behavior checks that frustrate scrapers frustrate a personal assistant trying to check out on a user's behalf. Canal's point, as reported, is that the infrastructure never learned to tell the difference.

The numbers behind the friction are stark. A robots.txt analysis of the internet's top 2,000 domains, published on Dev.to in August, found that of 1,530 reachable domains, 292 — about 19 percent — now disallow at least one of the four major AI crawlers. Amazon blocks all five major AI crawlers outright, alongside Instagram, X, Pinterest, and TikTok. Crawlers are not the same as task-performing agents, but the direction of travel is identical: the default posture of the commercial web toward automated visitors — websites blocking AI agents as a class — is shifting from tolerance to refusal.

For now, the agent makers are not winning the argument against the websites blocking AI agents. The Verge reported that OpenAI and Google did not respond to requests for comment on the roadblocks, and Instinct declined to comment. Meta's partnership strategy is the most developed counter-move, but partnerships are slow, bilateral, and fragile — one policy change at a partner retailer and the corridor closes. Until there is a standard way for a website to verify that an agent is acting for a real customer with that customer's permission, the web's doors will keep closing one by one. Every new report of websites blocking AI agents makes the permission layer — not the model — the binding constraint on the agent economy. The agents are getting smarter. The web is getting pickier about who it lets in.