Can ChatGPT, Claude, Gemini, and Perplexity crawl your website?

A concrete aperture with a narrow beam of light reaching the surface beyond it, representing crawler access

Before rewriting a page for AI, confirm that the relevant system can retrieve it. The check sounds basic, but several distinct failure modes can produce the same missing result.

Vendors operate multiple crawlers for different purposes. A rule that limits training may not limit search. A page can look complete in a browser while its important content is missing from the initial HTML. A canonical can point somewhere else. A Bing directive can leave a page eligible for ordinary search but unavailable for Copilot citation. This guide shows how to separate those failure modes.

The fastest test: compare HTML with the rendered page

Open the page normally and identify the facts an answer engine would need: product names, prices, service areas, specifications, comparisons, FAQs, and contact details.

Then inspect the original HTML response. You can use “View Source,” a command-line request, or a server-side testing tool. Search that response for the same facts.

If the facts appear only after JavaScript runs, a crawler that does not render JavaScript may not receive them.

In December 2024, Vercel and MERJ reported that GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Meta-ExternalAgent, and Bytespider did not execute JavaScript across nextjs.org, the Vercel network, and two job-board sites built on different stacks. Googlebot-based systems rendered JavaScript in the same research. Server-render every fact that matters to a sale.

Search, training, and user fetch are different jobs

A crawler matrix separating search discovery, training controls, and user-requested retrieval

Interpret an “AI bot blocked” badge by checking the bot’s specific purpose.

Bot token	Vendor	What it is for	What blocking it actually does
`OAI-SearchBot`	OpenAI	Finds and surfaces pages in ChatGPT search results	Removes the site from ChatGPT search answers after OpenAI processes the change, although navigational links may still appear
`GPTBot`	OpenAI	Collects content that may be used to train OpenAI’s generative foundation models	Signals that the site’s content should be excluded from that training use; ChatGPT search access is controlled separately
`ChatGPT-User`	OpenAI	Fetches pages for certain user actions in ChatGPT and Custom GPTs	A `robots.txt` block may have no effect because the request is user initiated; this token does not control ChatGPT search inclusion
`ClaudeBot`	Anthropic	Collects public web content that could contribute to model training	Signals that future site material should be excluded from Anthropic’s model-training datasets
`Claude-SearchBot`	Anthropic	Indexes and analyzes pages to improve Claude search results	Prevents Anthropic from indexing the content for search optimization, which may reduce visibility and accuracy in Claude search results
`Claude-User`	Anthropic	Retrieves pages in response to a Claude user’s request	Prevents retrieval for those user-directed requests and may reduce visibility in Claude’s user-directed web search
`Googlebot`	Google	Builds Google Search indexes used by Search features, including generative features	Restricts crawling for Google Search and can prevent the page’s content from being eligible as supporting material in Google’s AI search features
`Google-Extended`	Google	Controls use of Google-crawled content for Gemini model training and grounding in Gemini Apps and Vertex AI	Opts the content out of those uses; it has no effect on Google Search inclusion or ranking
`PerplexityBot`	Perplexity	Crawls pages so Perplexity can surface and link them in search results	Prevents the crawler from indexing the content for Perplexity search, reducing its opportunity to surface there
`Perplexity-User`	Perplexity	Fetches a page in response to a user’s question	A `robots.txt` block generally has no effect because Perplexity treats it as a user-requested fetch; network-level access controls may still block the request

The last column determines the practical effect. Record the exact bot token you blocked; “AI bots blocked” leaves the policy undefined.

robots.txt is an instruction system, not a technical access-control boundary. User-requested agents may require network-level controls, and those controls should verify published IP ranges rather than trust a user-agent string alone.

Vendor documentation checked August 15, 2026. Bot names and behavior can change, so verify the linked documentation before changing access policy.

A sample robots.txt policy

The right policy depends on your goals. This example allows search discovery while declining OpenAI training collection:

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

Review every token and purpose before adopting a generic “allow all AI” block. Keep a dated policy record so future teams know why each decision was made.

Check canonicals in the original HTML

A canonical tells indexing systems which URL represents a page. On a client-rendered application, the canonical may be injected after load. A non-rendering crawler can miss it.

For each important page:

confirm a single canonical is present in the initial HTML;
confirm it uses the correct absolute URL;
remove conflicting canonicals from headers or markup;
avoid canonicalizing valuable detail pages to a generic category page;
confirm redirects resolve to the canonical destination.

Search engines may choose a different canonical when other signals conflict, so reinforce the tag with consistent internal links, redirects, and sitemap entries.

Bing indexing can affect ChatGPT and Copilot discovery

OpenAI has publicly described Bing-supported discovery for ChatGPT Search. Bing also powers Microsoft search experiences. If a page is absent from Bing’s index, it may be unavailable to a retrieval path even when Google indexes it.

Check a representative set of important URLs in Bing Webmaster Tools or Bing search. If they are missing, inspect crawl errors, redirects, canonicals, content quality, and sitemap accuracy.

Bing eligibility is a meaningful discovery check for ChatGPT Search and Microsoft surfaces. Record it as one retrieval path because platform behavior and search partners can change.

Watch NOARCHIVE and NOCACHE

Bing’s webmaster guidance says NOARCHIVE and NOCACHE can prevent content from being used as a citation in Copilot while leaving it available in ordinary search results.

That makes these directives easy to miss. They may appear as HTML meta tags or response headers. Audit both.

If the directive exists for a legal, licensing, or content-control reason, preserve it until the responsible owner reviews the tradeoff.

Keep the sitemap boring and accurate

A sitemap should list canonical, indexable, successful URLs. Remove redirects, duplicates, and deleted pages.

In the same December 2024 dataset, Vercel and MERJ observed 34.82% of ChatGPT crawler requests and 34.16% of Claude crawler requests hitting 404s. Your site’s rate will differ, but stale URLs still waste requests and obscure which pages matter.

Audit for:

non-200 responses;
redirected URLs;
non-canonical URLs;
pages blocked by robots or noindex;
URLs from retired site sections;
missing high-value pages.

Spend sitemap time on live canonical URLs. No major AI engine documents priority or changefreq as visibility levers.

What the evidence says about llms.txt

An llms.txt file is inexpensive to publish and may be convenient documentation for humans or experimental tools. No major answer engine has confirmed that the file improves citation eligibility.

Independent analyses have found little or no crawler use and no measurable citation effect. If you publish one, classify it as a low-cost experiment or a human-readable documentation aid.

Never let llms.txt distract from initial HTML, crawler policy, indexability, canonicals, and live URLs. Those have documented mechanisms.

The technical audit checklist

For every page containing a fact that affects a sale:

Passing this checklist establishes retrieval eligibility. Citation and recommendation depend on the query, source selection, and the system producing the answer.

Next: use the evidence-backed AEO checklist to decide what to fix after access is confirmed.