Is Cloudflare blocking search crawlers?
robots.txt declares access preferences; security rules can still block requests. Separate search, training and user-triggered crawlers, then identify the rule that acted on the request.
Choose the crawler’s purpose
Search, training and user-triggered requests serve different purposes. OpenAI documents OAI-SearchBot, GPTBot and ChatGPT-User separately. Match your chosen access policy to the crawler’s role.
Compare responses with security events
Compare a normal request with one carrying the selected bot user-agent. A difference is evidence to investigate; it does not prove how a verified crawler’s real request is treated.
curl -I -A "OAI-SearchBot" https://yourdomain.com/
Find the rule that acted
Review AI bot preferences in Cloudflare Security Settings and crawler events in AI Crawl Control. Evaluate the relevant bot or WAF rule against your intended policy. Loading a page in a browser does not prove crawler access.
Check again after the change
Recheck the status, returned content and security event. Removing a block does not guarantee recrawling or inclusion in AI answers.
Frequently asked questions
No Disallow in robots.txt but bots are still blocked — normal?
Yes, and it's the most common hidden problem — robots.txt is only a declaration; edge layers like Cloudflare block independently of it.
I don't use Cloudflare — can the same problem exist?
Yes — other CDNs, hosting WAFs and CMS security plugins like Wordfence produce similar blocks. The verification command is the same.
Once I remove the block, do I appear in AI immediately?
Not immediately — bots need to rediscover your site and engines need to process the content. Removing the block is the prerequisite, not a results guarantee.
These guides explain technical checks. Passing them does not guarantee search ranking or inclusion in AI answers.