ChatGPT can't see my site: why AI misses it
The four most common technical reasons for invisibility in AI answer engines: your bot is blocked at the edge (even with a permissive robots.txt), robots.txt explicitly disallows search bots, your pages look empty without JS, or structured data is missing. These are measurable and fixable — but know this: technical accessibility is the prerequisite for being cited, not the guarantee.
Two separate gates: policy vs real access
robots.txt is your site's declaration — it says "I don't want these bots". But the real gate lives elsewhere: a WAF, bot-protection service or CDN layer can block bot requests independently of robots.txt. The owner looks at robots.txt and assumes "we're open"; in reality OAI-SearchBot gets a 403.
This distinction matters because the diagnosis differs: you see the declaration by opening the file, but the real block only shows when you send a request as the bot itself. Verify yourself: curl -s -o /dev/null -w "%{http_code}" -A "OAI-SearchBot" https://yourdomain.com/ — if it returns 403/503 instead of 200, the problem is at the edge, not in robots.txt.
What do you lose by blocking each bot?
Bots come in three classes with different consequences:
- Search/citation bots (OAI-SearchBot, Claude-SearchBot, PerplexityBot): blocking these directly cuts your visibility in answer engines.
- Training bots (GPTBot, ClaudeBot, CCBot): blocking limits your content's use in model training — a legitimate licensing choice with indirect visibility impact.
- User-fetch bots (ChatGPT-User): runs when a user tells the AI "look at this page"; block it and your site can't be read ad-hoc.
Rather than "block everything" or "allow everything", keeping search bots open is usually the right balance.
The bot arrives but can't read: JS shells and structured data
Most AI crawlers don't run JavaScript — if your page doesn't carry real text in the first HTML, the bot sees an empty shell. Check the source (Ctrl+U): is your content there as text?
The second layer is structured data: JSON-LD markup for Product, Organization, FAQ lets the machine answer "what is this, whose is it, what does it sell" with confidence. If the source has no application/ld+json, the answer engine struggles to classify the page.
Technically clean but still invisible: missing evidence
Answer engines don't evaluate a site in isolation — they look for independent sources mentioning you: directories, reviews, forums, news. If you're technically flawless but nothing on the web verifies you, the model says "I don't know this business". This layer can't be measured from outside and isn't fixed by one scan — it takes original data, customer reviews and industry mentions.
Also, each engine's source ecosystem differs: appearing on ChatGPT doesn't mean appearing on Perplexity — measurement must be done engine by engine.
Frequently asked questions
Is there a tool/consultant that guarantees I'll appear in AI?
No — anyone promising that is selling marketing. What can be done is removing blockers and increasing citability; the decision belongs to the engine.
Will adding llms.txt make me visible?
Not by itself — llms.txt is a bridge introducing your site to agents; if there's a block or content can't be read, it changes nothing. On a clean site it's good practice.
No blocks on my site but competitors show up, I don't
Then the problem isn't technical — it's the authority/evidence layer: independent sources talk about your competitor more. Compare your forum, directory and review presence.
These guides explain technical checks. Passing them does not guarantee search ranking or inclusion in AI answers.