Should you block AI agents from your website? What Amazon's fight with Muse means for a small business

On September 8, Meta launched Muse, a personal AI agent that shops and books on its owner’s behalf. By September 21, Amazon had shut it out. Anyone who asked Muse to buy something on Amazon got a notice instead: continued access by an unauthorized AI agent violates Amazon’s Conditions of Use. Amazon had asked Meta to keep the agent off its site, and Meta had declined.

Amazon’s spokesperson put the company’s position plainly: third-party applications that buy on a customer’s behalf “should operate openly and respect service provider decisions,” and should identify themselves the way a delivery app or a travel agency does. Amazon also said that Muse doesn’t identify itself when it browses. Meta’s reply focused on security, saying that credentials are held in the user’s own secure storage and never exposed to the model.

Amazon can afford that fight, because it sells more through its own channels than any agent will send it. For a dental practice, a plumber or a boutique, the math runs the other way. Yet a lot of small businesses are doing what Amazon did without ever deciding to, and this post is about how to make that decision.

Why Amazon’s block is different from yours

Amazon blocked Muse on purpose, with a policy, and it told Meta first. When a small business blocks an agent, it’s almost always a side effect of a setting that someone turned on for a different reason, such as a bot-fighting mode on the CDN, a CAPTCHA in front of the booking page, or a firewall rule that challenges anything from a datacenter address.

Those settings don’t know the difference between a scraper and a customer’s assistant trying to book a cleaning. Muse in particular runs a real Chromium browser inside a cloud machine assigned to each user. It doesn’t announce a crawler name, Meta hasn’t published one, and robots.txt rules don’t apply to it. From your website’s side it looks like a person on a datacenter connection, and aggressive bot protection is built to turn that kind of visitor away.

In a first trial run of our scanner on October 1, two of fourteen Austin dental and HVAC websites met a plain browser request with a bot wall, and about a quarter of the booking pages we could reach put a CAPTCHA in front of the form. That’s a small sample and we’ll publish the full census, but it matches what you’d expect from default settings on common platforms.

Why one rule for every AI bot is a mistake

Most accidental blocking comes from treating “AI bots” as one thing. There are three kinds, and each one needs its own decision.

Training crawlers. GPTBot (OpenAI), ClaudeBot (Anthropic), Google-Extended (Google’s Gemini training and grounding), Meta-ExternalAgent. These collect content for building models. Blocking them in robots.txt does not remove you from any search or assistant. OpenAI’s own documentation says that disallowing GPTBot prevents your content from being used in training, and it says nothing about search. If you don’t want your pages in training data, you can block these and nothing else changes.

Search and retrieval crawlers. OAI-SearchBot (ChatGPT search and shopping), Bingbot (which also feeds ChatGPT and Copilot), Googlebot, Claude-SearchBot, PerplexityBot. These make up the index that the assistants draw from when a person asks for a dentist nearby. OpenAI’s documentation is explicit: sites that opt out of OAI-SearchBot will not be shown in ChatGPT search answers. If you block these, you’ve opted out of being recommended.

User-triggered fetchers and agents. ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, and the browser-based agents like Muse. These visit your site because a specific person asked their assistant to look at it or book with it. OpenAI notes that because these actions are initiated by a user, robots.txt rules may not apply, and for Muse there’s no name to list at all. The only thing that governs these visitors is your bot protection.

So for a business that wants customers, the sensible default is to decide about training on its own merits, allow search and retrieval, and make sure that your bot protection lets user-triggered agents reach the pages they need.

What to do

Read your robots.txt as three separate lists. Many hosts and plugins added blanket AI blocks in 2024, so go through yours bot by bot. If OAI-SearchBot or Bingbot is disallowed, you won’t show up in ChatGPT’s recommendations, whether you meant that or not.

Find out what your bot protection does before you loosen it. Check the CDN or host dashboard for a bot-fighting mode, a managed challenge rule, or a datacenter-traffic rule. Look at the security log for challenged or blocked requests to your booking or checkout pages. If there are a lot of them, some will be agents and some will be the scrapers you wanted to stop, and you want to tell the two apart instead of letting everything through.

Exempt only the pages an agent needs. That means pages like the booking page, the pricing page and the policies page, while the protection stays on everything else. If a CAPTCHA has to exist, keep it off the first screen and put it at the final confirmation step, so that an agent can at least read availability and report it to its owner.

Prefer verification to challenges. Cloudflare’s bot management now sorts verified bots into categories (search, agent, training, transact among them), and under the Web Bot Auth approach it can verify an agent’s identity with a cryptographic signature, so the agent doesn’t have to solve a puzzle. Cloudflare says all customers can configure AI bot policies to set their own allow and block expectations. That’s the approach we’d recommend: let agents that identify themselves through, and make a deliberate choice about the ones that don’t.

Decide about Muse specifically. Because Muse doesn’t identify itself, you can’t allow it by name. You can only decide whether datacenter browser sessions that behave like customers get through. For a business that takes bookings, the answer is usually yes for the booking path, with rate limits in place and no wall.

Caveats

Amazon’s concern about agents that don’t identify themselves is a real one, and it isn’t only about Amazon’s ad revenue. An agent that browses as a logged-in person, with stored credentials, is a new kind of visitor, and the standards for how it should announce itself are still being settled. Web Bot Auth is one answer, and Meta hasn’t said whether Muse will adopt it.

Loosening bot protection also has a cost, because scrapers and credential stuffers use the same datacenter addresses that agents do. That’s why our advice is to exempt specific pages and prefer verification, and not to turn protection off.

We can say with confidence that the businesses blocking agents today are mostly doing it by accident, and that fixing it takes a decision and doesn’t require a rebuild. If you’d like to know whether your site is one of them, the free assessment checks for that from a datacenter browser, the way an agent would see your site.