Are You Accidentally Blocking ChatGPT From Your Website? AI Crawlers and robots.txt, Explained
June 22, 2026
If a customer asks ChatGPT or Perplexity to recommend a business like yours and your name never comes up, the cause might not be your reviews, your website, or your competitors. It might be a single line of text most owners have never seen: your robots.txt file. Plenty of small-business sites quietly tell AI engines to stay out — usually because a developer added the rule back in 2023, when "block the AI bots" was common advice. Two years later those same engines are recommending businesses to real customers, and that old rule is keeping you off the list.
What robots.txt is, in plain English
Every website has (or can have) a small file at yourdomain.com/robots.txt. It is a set of instructions for automated visitors — "crawlers" — telling them which pages they are allowed to read. Search engines and AI engines check it before they look at your site. A friendly rule waves them in; a Disallow rule turns them away. The file is public, it is easy to get wrong, and most owners have never opened it.
Training bots and answer bots are not the same thing
This is the part that trips people up. The major AI companies now run separate crawlers for two very different jobs, and you can allow or block each one independently:
- Training crawlers collect text to help build future AI models. These include GPTBot (OpenAI), ClaudeBot (Anthropic), and Google-Extended (Google). Blocking these keeps your content out of model training — a perfectly fair choice.
- Answer crawlers fetch live pages when a person actually asks a question, so the AI can cite a current, real source. These include OAI-SearchBot and ChatGPT-User (OpenAI), PerplexityBot (Perplexity), and Claude-SearchBot and Claude-User (Anthropic). These are the ones that decide whether you can be quoted when a customer asks right now.
Here is the catch: blocking GPTBot does not remove you from ChatGPT's answers, because answers come from a different bot (OAI-SearchBot). And one reassurance worth repeating — blocking Google-Extended does not hurt your normal Google Search ranking at all. The two are completely separate systems.
The mistake that makes you invisible
The damage usually comes from a well-meaning blanket rule — something a template, a plugin, or a developer added to "block AI" — that disallows every AI user-agent, including the answer bots. The result is silent: your site loads fine, Google still works, but when a customer asks Perplexity for "a good [your trade] near me," your pages cannot be read, so you cannot be cited. You never see an error message. You just never come up.
How to check your own site in five minutes
- Type your domain followed by
/robots.txtinto a browser — for example,yourbusiness.com/robots.txt— and press enter. - Look for any
Disallow: /line sitting under an AI user-agent such asOAI-SearchBot,PerplexityBot,ChatGPT-User, orClaude-SearchBot. That line is blocking that engine. - Watch for a catch-all
User-agent: *followed byDisallow: /, which blocks essentially everything. - If you see nothing about AI bots at all, that is usually fine — most engines are allowed by default unless they are told otherwise.
- Not sure what you are reading? Send the file to whoever maintains your site and ask plainly: "Are we blocking any AI search crawlers?"
What to do if you find a block
If your goal is to be found in AI answers, you want the answer bots allowed. You can still block the training crawlers if you would rather not feed model training — that choice has no effect on whether you get cited today. Ask your web person to remove the Disallow rules for OAI-SearchBot, ChatGPT-User, PerplexityBot, and the Claude answer bots, or simply to allow all crawlers if you have no reason to restrict them. It is a small edit, and it can be live within minutes.
robots.txt is only the doorway, though. Clearing it means an AI engine can read your site; being recommended still depends on clear pages, consistent business details, and credible mentions elsewhere. If you would rather not decode any of this by hand, EchoRank checks whether the major AI engines can actually see and cite your business and shows you exactly where the gaps are. You can run a free audit and see what ChatGPT, Perplexity, and Gemini know about you today.