Can ChatGPT see your website? A plain-English check for business owners

Short answer: Possibly not. For ChatGPT and other AI assistants to recommend your business, their crawlers must be allowed in by your robots.txt, must get past any bot blocking on your CDN or firewall, and must find your actual words in the page's HTML rather than in JavaScript they don't run. Your pages also need to be indexed in Google and Bing, because AI answers lean heavily on search indexes. Each of these can be checked in a few minutes, and the checklist below shows you how.

We sell SEO and answer engine optimisation (AEO), so we have an interest here. Everything below can be done without us, and we link to the official documentation for each point.

Why does this matter more than schema or llms.txt?

Because access comes first. If a crawler can't read a page, nothing else on that page counts. Google says that to appear in AI Overviews or AI Mode, a page simply needs to be indexed and eligible to show a snippet, with "no additional technical requirements" (Google Search Central).

By contrast, the popular quick fixes show little effect. Ahrefs tracked pages that added structured data and found no meaningful rise in ChatGPT citations (Ahrefs). In a separate study it found that most llms.txt files are never requested by AI bots at all (Ahrefs). Both are fine as housekeeping. Neither will fix a site that crawlers can't get into.

Which AI crawlers should you know about?

Each AI company runs several bots, and they do different jobs. The key distinction is between bots that train models and bots that fetch pages for answers and search results. Blocking one doesn't block the other.

Sources: OpenAI's crawler documentation, Perplexity's crawler documentation, Anthropic's crawler policy and Google's list of common crawlers. Anthropic also runs separate bots for user requests and search; its page lists them.

What we recommend for most service businesses: allow the search and user-request bots (OAI-SearchBot, ChatGPT-User, PerplexityBot, and Googlebot and Bingbot of course). Whether to allow the training bots (GPTBot, ClaudeBot, Google-Extended) is a policy choice about your content, not a visibility fix. Many businesses that want to be recommended are happy to allow them. Others aren't, and that's a legitimate decision.

How do you check your robots.txt?

Type your domain followed by /robots.txt into a browser, for example yourcompany.com/robots.txt. You're looking for lines like these:

User-agent: OAI-SearchBot
Disallow: /

That pair blocks OpenAI's search crawler from the whole site. Disallow: / under User-agent: * blocks every bot that doesn't have its own rule, which is worse. An empty Disallow: line, or no rule at all for a bot, means it's allowed.

Common surprises we see:

• A blanket block left over from when the site was in development.
• A plugin or theme setting ("discourage search engines") that was never switched back.
• A copy-and-paste "block all AI bots" list that included search bots along with training bots.

Could Cloudflare or your firewall be blocking AI bots?

Yes, and robots.txt won't show it. Many sites sit behind a CDN or web application firewall (WAF) such as Cloudflare, and these can refuse bots before they ever reach your robots.txt. In July 2025 Cloudflare announced it was changing its default to block AI crawlers unless they pay for content (Cloudflare). Cloudflare's bot settings (documentation) let you decide which bots get through.

If your site uses Cloudflare, ask whoever manages it to check:

• whether any "block AI bots" or AI crawl control setting is switched on;
• whether bot fight mode or custom firewall rules challenge verified bots;
• whether the bots you want, especially OAI-SearchBot and PerplexityBot, are allowed. Perplexity's documentation explains how to allow its bot through a WAF.

This is the check most often missed, because the site looks perfectly normal to people.

Does JavaScript stop AI from reading your site?

Often, yes. Many modern sites are built so that the browser downloads a near-empty page and then fills it in with JavaScript. Google renders JavaScript, so Google usually copes (Google Search Central). Most AI crawlers don't. Vercel analysed AI crawler traffic across its network and found that none of the major AI crawlers it looked at, including OpenAI's bots, ClaudeBot and PerplexityBot, rendered JavaScript (Vercel). Gemini was the exception, because it uses Google's infrastructure.

The test: open a page on your site, right-click and choose "View page source" (not "Inspect"). Search the source for a sentence you can see on the page. If it isn't there, AI crawlers probably can't see it either.

The fix is server-side rendering or pre-rendering, so the HTML arrives with the words already in it. Most frameworks support this. We rebuilt our own site this year for exactly this reason: it had been a JavaScript-only app, and AI crawlers saw an almost empty page.

Why do Google and Bing indexing matter for ChatGPT?

Because AI assistants don't crawl the whole web live for every question. They lean on search indexes. ChatGPT search combines OpenAI's own crawler with results from outside search providers, so being well indexed in Bing as well as Google is a sensible baseline. Microsoft's Copilot and Bing AI answers draw directly on Bing's index. In February 2026 Microsoft added an AI Performance report to Bing Webmaster Tools that shows when your pages are cited in Copilot and partner AI experiences (Bing Webmaster Blog). Google's AI Overviews and AI Mode use Google's own index.

Many businesses have set up Google Search Console and never touched Bing. Doing both takes an afternoon:

• Google Search Console: verify your site, submit your sitemap, and check the Pages report for pages that are "not indexed".
• Bing Webmaster Tools: verify your site (you can import it from Search Console), submit your sitemap, and turn on IndexNow if your platform supports it so that Bing hears about changes quickly.
• Local businesses: keep your Google Business Profile and Bing Places listing accurate.

A quick spot check: search site:yourcompany.com in both Google and Bing. If important pages are missing from either, AI assistants that rely on that index are unlikely to cite them.

A 20-minute self-check

1. robots.txt: open /robots.txt. Is there any Disallow: / under *, OAI-SearchBot, ChatGPT-User, PerplexityBot, Googlebot or Bingbot?
2. Training bots: have you made a deliberate decision about GPTBot, ClaudeBot and Google-Extended, rather than inheriting one?
3. CDN or firewall: if you use Cloudflare or similar, are AI bot blocking or challenge rules switched on? Are the bots you want allowed?
4. View source: is the text of your homepage and your main service pages in the raw HTML?
5. Google: is your site verified in Search Console, with a sitemap submitted and key pages indexed?
6. Bing: is your site verified in Bing Webmaster Tools, with a sitemap submitted? Does site: show your key pages?
7. Ask the assistants: ask ChatGPT, Perplexity and Gemini a question a buyer would ask, such as "best [your service] in [your city]". Are you mentioned? Who's named instead? Repeat it a few times, because answers vary.

If steps 1 to 6 pass and you still don't appear in step 7, the problem is no longer access. It's usually that other sites mention your competitors more, or that your pages don't answer the questions buyers ask. That's a content and reputation job, not a technical one.

Frequently asked questions

Will blocking GPTBot remove my business from ChatGPT?

No. OpenAI says GPTBot is used for training only. Visibility in ChatGPT search depends on OAI-SearchBot, which you control separately in robots.txt.

Does blocking Google-Extended hurt my Google rankings?

No. Google says Google-Extended doesn't affect inclusion or ranking in Google Search. AI Overviews follow Googlebot's rules.

Do I need an llms.txt file or special schema for AI?

Neither is required. The evidence so far shows little effect on AI citations from either. Keep accurate structured data for rich results in Google, but put crawler access and indexing first.

My site looks fine in a browser. How can AI not see it?

Browsers run JavaScript, and most AI crawlers don't. Firewalls can also block bots while letting people through. Use "View page source" and check your CDN's bot settings.

How long until changes show up in AI answers?

There's no fixed timescale, and nobody can guarantee a citation. Fixing access lets crawlers read your pages on their next visit. Submitting sitemaps and using IndexNow helps search indexes catch up sooner.

Want us to check for you?

Our free AI visibility check looks at your crawler access, rendering and indexing, and asks real buyer questions across the main AI assistants to see whether you're mentioned. You'll get a short, plain report with no obligation. If you'd like ongoing help, see our pricing.