How AI crawlers like GPTBot & ClaudeBot work

Here's a problem plenty of brands don't know they have: they're working hard to show up in AI answers while quietly blocking the very bots that would put them there. Before ChatGPT, Perplexity or Claude can mention your site, a crawler has to fetch and read it — and thanks to overzealous firewalls, default bot-blocking and a robots.txt someone edited two years ago, a surprising number of sites turn those crawlers away at the door without realising it.

So let's meet the bots, learn to spot them in your logs, and make sure you're rolling out the welcome mat instead of the bouncer.

Two jobs, not one

The single most useful thing to understand is that "AI crawler" covers two very different activities, and you may feel differently about each. Some bots crawl the open web to gather training data for future model versions — that's the one publishers argue about, and the one you can opt out of without hurting yourself much. Others fetch your page in real time because a user just asked a question your site can answer; these "retrieval" or "search" bots are the ones that decide whether you appear in today's answers. Blocking a training bot is a strategy choice. Blocking a retrieval bot is shooting yourself in the foot.

The main AI crawlers, and what each is for

Crawler (user-agent)	Who	Job
GPTBot	OpenAI	Training data collection
OAI-SearchBot	OpenAI	Indexing for ChatGPT search
ChatGPT-User	OpenAI	Live fetch when a user's prompt needs your page
ClaudeBot	Anthropic	Training / retrieval for Claude
Claude-User	Anthropic	Live fetch during a Claude conversation
PerplexityBot	Perplexity	Indexing for Perplexity answers
Google-Extended	Google	Controls use in Gemini / AI training (not normal Search)

The nuance worth remembering: Google-Extended only governs AI use — blocking it does not remove you from normal Google Search. And the "-User" bots (ChatGPT-User, Claude-User) fire the moment a real person's question needs your page, which is about as high-intent as web traffic gets.

How to see who's actually visiting

Everything a crawler does shows up in your server logs as a request with its user-agent string. If you can read your access logs — through your host, a Cloudflare/analytics dashboard, or a log file — search for the names above (GPTBot, ClaudeBot, PerplexityBot, and so on). Their presence tells you the engines are reading you; their absence, when you'd expect them, is a red flag that something is turning them away. If digging through raw logs isn't your idea of fun, GEOpta's AI Crawler Analytics surfaces which bots are hitting your site, how often and on which pages, without the grep.

The mistake almost nobody notices: blocking them

This is where most AI visibility quietly dies. Three common culprits:

• robots.txt — a Disallow rule (sometimes added long ago, sometimes by a plugin, sometimes copied from a template) that names GPTBot or blocks everything.
• A firewall or WAF rule — security tooling that treats unfamiliar bots as threats and returns a 403.
• Cloudflare's bot-fight / "block AI bots" toggles — a single well-meaning switch in the dashboard that shuts the door on the exact crawlers you want.

Any one of these can make you invisible to AI while your site looks perfectly healthy to humans. It's worth an explicit audit, because the failure is silent — nothing breaks, you just never show up.

How to let them in (the right ones)

The fix is mostly about not blocking. Make sure your robots.txt allows the retrieval bots you care about, confirm your WAF or Cloudflare settings aren't quietly 403-ing them, and don't hide behind an aggressive "block all bots" posture. If you want to opt out of training while staying visible in answers, that's a legitimate middle path — allow the retrieval and search bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) while disallowing the pure training crawlers (GPTBot, Google-Extended) in robots.txt. Just go in with eyes open: block the retrieval bots and you're opting out of AI visibility entirely.

The JavaScript trap

Even when you welcome them, there's a catch: most AI crawlers don't run JavaScript. If your content is rendered client-side — loaded in after the page arrives — a bot that reads the raw HTML sees an empty shell. That's why server-side rendering or an edge layer matters so much for AI visibility. GEOpta's Cloudflare worker exists partly for this reason: it injects your structured data and serves a clean, extraction-ready version of your pages at the edge, so the crawlers that skip JS still get the substance. More on the on-page side in optimizing your website for AI and the llms.txt guide.

Bottom line

You can't be recommended by an AI that can't read you. Learn the bots, check your logs to confirm they're getting through, and audit robots.txt, your WAF and your Cloudflare settings so you're not blocking the retrieval crawlers by accident. Decide deliberately about training bots if you like — but keep the answer bots welcome, and make sure your content survives without JavaScript. Not sure who's visiting? A free AI Visibility Score and GEOpta's crawler analytics will show you.

Frequently asked questions

What are AI crawlers like GPTBot and ClaudeBot?

They're bots that AI companies use to read the web. Some, like GPTBot and Google-Extended, mainly collect training data for future models. Others, like OAI-SearchBot, ChatGPT-User, PerplexityBot and Claude-User, fetch pages to help answer a user's question in real time. The retrieval bots are the ones that determine whether your site appears in AI answers, so they're the ones you most want to allow.

Am I accidentally blocking AI crawlers?

It's common. The usual culprits are a robots.txt Disallow rule, a firewall or WAF that returns a 403 to unfamiliar bots, or a Cloudflare 'block AI bots' toggle. Any of these can make you invisible to AI while your site looks fine to humans. Check your access logs for user-agents like GPTBot, ClaudeBot and PerplexityBot — if they're absent when you'd expect them, something is turning them away.

Should I block AI crawlers or allow them?

Allow the retrieval and search bots if you want to appear in AI answers — blocking them opts you out of AI visibility entirely. You can take a middle path: allow the answer bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, Claude-User) while disallowing pure training crawlers (GPTBot, Google-Extended) in robots.txt, if you'd rather not contribute training data. Note that Google-Extended only affects AI use, not normal Google Search.

Do AI crawlers run JavaScript?

Most don't. If your content is rendered client-side with JavaScript, a crawler reading the raw HTML sees an empty shell and misses your substance. That's why server-side rendering or an edge layer matters for AI visibility — it ensures the crawlers that skip JS still receive your actual content and structured data. GEOpta's Cloudflare worker addresses this by injecting schema and serving an extraction-ready version of your pages at the edge.

How do I see which AI bots are visiting my site?

Look in your server access logs for the crawler user-agent strings such as GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot and Google-Extended. Their presence confirms the engines are reading you; their absence is a warning sign. GEOpta's AI Crawler Analytics automates this, showing which bots hit your site, how often and on which pages, so you don't have to parse logs by hand.