23% of UK SME Websites Are Blocking AI Crawlers in Their robots.txt

Last updated: March 2026

We checked the robots.txt files of 30 UK small business websites to see whether they were blocking AI crawlers like GPTBot, ClaudeBot or PerplexityBot. Nearly a quarter were.

---

TL;DR

• 7 out of 30 UK SME websites (23.3%) block at least one AI crawler
• GPTBot (used by ChatGPT) is the most commonly blocked: 7 sites (23.3%)
• ClaudeBot (used by Anthropic's Claude) is blocked by 5 sites (16.7%)
• PerplexityBot is blocked by just 1 site (3.3%)
• In several cases, the same sites had active SEO signals suggesting they want to be found online — raising the question of whether the blocks are intentional
• 93% of sites had a robots.txt file in place, but the AI crawler rules within them varied considerably

---

What is robots.txt and how does it affect AI visibility?

robots.txt is a plain text file at the root of a website that tells web crawlers which parts of the site they can and cannot access. It has been a standard part of the web since the mid-1990s, originally designed to manage search engine crawlers like Googlebot and Bingbot.

In recent years, AI companies have introduced their own crawlers. OpenAI uses GPTBot to gather content for ChatGPT. Anthropic uses ClaudeBot for Claude. Perplexity uses PerplexityBot. These crawlers read website content to help their AI systems understand what businesses do, what services they offer, and what information is on their pages.

If a website's robots.txt file includes a rule that blocks one of these crawlers, that AI platform cannot access the site's content through crawling. This does not necessarily mean the business will never appear in that platform's responses — AI systems draw from multiple sources beyond direct crawling — but it does remove one significant channel through which the platform can learn about the business.

---

What we found

AI crawler blo