Calgary, AB

Which AI crawlers to allow, and what blocking them costs.

Most sites blocking AI crawlers never decided which kind to block. Live-search access and model-training permission are different choices, but many plugins collapse them into one switch.

Last reviewed

Check your site free →
On this page7 sections

How do you check what your site currently allows?

Open yoursite.com/robots.txt in a browser. Look for any Disallow rule under a user-agent whose name you recognise as an AI crawler.

It's a plain text file, it's public on every site, and it takes thirty seconds to read. A Disallow: / under a named bot blocks that bot entirely from your site.

Also check for a blanket User-agent: * / Disallow: /, which blocks everything including Google. That one is usually a staging site that went live without the file being changed back, and it's more common than you'd think.

What does each AI crawler actually do?

They split into training crawlers and live-answer crawlers, and blocking the second kind is what removes you from AI recommendations.

This distinction is the single most misunderstood thing about AI crawler policy, and getting it backwards is common — a lot of plugins block the retrieval bots while allowing the training ones, which is precisely the wrong way round for a business that wants to be recommended.

GPTBot
— OpenAI's training crawler. Blocking it keeps your content out of model training and does not affect live ChatGPT answers.
OAI-SearchBot
— Fetches pages to answer a live ChatGPT search. Blocking this is what makes you uncitable in ChatGPT.
Claude-SearchBot
— Anthropic's live-search crawler. Blocking it can reduce visibility in Claude's web-search results; ClaudeBot is the separate training crawler.
PerplexityBot
— Perplexity's crawler. Perplexity shows its sources visibly, so being allowed is most obviously valuable here.
Google-Extended
— Controls Gemini model training and grounding in Gemini apps. It does not control Google Search or AI Overviews and is not a ranking signal.

Should a local business block any of them?

Allow live-search crawlers if you want to be cited. Decide separately whether training crawlers may reuse your content.

OAI-SearchBot, Claude-SearchBot and PerplexityBot are the visibility decision: blocking them can reduce whether your pages appear in sourced answers. GPTBot and ClaudeBot are the training decision, and opting out of those does not by itself remove you from live search.

A local service business will usually want live-search access because the content describes a service someone has to hire locally. Training permission is a separate policy choice, not an SEO requirement.

Check my site

The same checks this page describes, run against your own site. About 30 seconds.

Run the free check →

What should the robots.txt say?

Allow everything, point at your sitemap, and only disallow genuinely private paths. For most local businesses that's a four-line file.

The working version for a local business is: User-agent: *, Allow: /, a Disallow: line for any admin or private path, and a Sitemap: line with the full URL of your sitemap. No named AI crawler needs its own rule if you're allowing everything.

If you do want to be selective — allowing retrieval bots while blocking training crawlers — name them explicitly rather than relying on a plugin's preset, because presets change without telling you.

Why does blocking happen by accident?

Security plugins, CDN bot-protection defaults and 'AI protection' toggles that were switched on without anyone reading what they did.

WordPress security and SEO plugins have added AI-blocking options over the last couple of years, several of them on by default. CDN and hosting providers have added bot-protection tiers that block unknown user agents wholesale, which catches AI crawlers along with scrapers.

The problem is that 'protect your content from AI' rarely says whether it blocks training, live retrieval, or both. Read the actual user-agent rules instead of trusting the switch label.

Is robots.txt enough on its own?

No. Access is necessary and not sufficient — the crawler still has to find something readable when it arrives.

The second half of the problem is a site that renders entirely in JavaScript. Google usually copes; several AI crawlers fetch the HTML, find an empty shell and leave. Disable JavaScript in your browser, reload your homepage, and what remains is roughly what they see.

After that it's the ordinary work: clear answer-first passages, accurate business details and useful pages that can stand as sources. An optional llms.txt can index those pages for tools that choose to request it, but major search providers do not require it.

How do you tell whether it's working?

Ask the assistant the questions your customers would ask and see whether you're named. Check your server logs for the crawler user agents.

The direct test is the honest one: put the question a customer would ask into Perplexity and read what comes back. Keep the question list stable so month-to-month comparison means something.

Server logs tell you whether the crawlers are actually visiting, which separates 'not being cited' from 'not being read'. Those are different problems with different fixes.

Common questions

Will allowing AI crawlers hurt my Google ranking?

No. Google-Extended is a separate control for Gemini training and grounding. It does not control inclusion or ranking in Google Search, including AI Overviews.

Do AI crawlers respect robots.txt?

The major named ones publish their user agents and state that they honour it. Scrapers that ignore robots.txt exist, but they aren't the ones producing recommendations.

Should I add an llms.txt too?

Only if you want a cheap optional index for tools that choose to read it. It is not required, Google Search says it does not use it, and the free checker does not score it.

How do I check without reading robots.txt myself?

The free check on https://brainwage.ca tests exactly this and reports which crawlers are blocked, in about thirty seconds.

See what's broken before you spend anything.

The free check takes about 30 seconds and asks for nothing. From $500 a month if you want us to fix what it finds.

Run the free check →

Related