Something's wrong
Did my website accidentally block AI crawlers?
Answered by Ryan, RankNext strategist · Updated July 2, 2026
The short answer
Very possibly — it's one of the most common self-inflicted wounds we find. Copy-pasted robots.txt templates, security plugins, and CDN bot protection routinely block GPTBot, PerplexityBot, ClaudeBot, and Applebot without the owner ever deciding to. Check your robots.txt and bot settings, or run a free scan that reads your site like the engines do.
How the accidental block happens
Three usual paths. One: a robots.txt template copied from a blog post that disallowed 'AI bots' wholesale back when that was fashionable. Two: a WordPress security plugin or firewall preset that classifies answer-engine crawlers as scrapers. Three: CDN bot protection tuned aggressively enough that GPTBot gets a challenge page instead of your homepage.
The cruel part is the silence. Nothing on your site looks broken, your Google rankings don't change, and meanwhile every AI engine that customers increasingly ask has quietly stopped being able to read you — so the answers name whoever it can read.
The two-minute check, and the fix
Open yoursite.com/robots.txt and look for user-agents like GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot, Applebot, and Applebot-Extended under a Disallow. Then check any security or CDN dashboard for 'AI scraper' toggles. Our free AEO checker does this automatically and also verifies your pages render readable HTML.
The fix is a deliberate allowlist: name the engine crawlers you want (for a local business wanting customers from AI, that's all of them) and keep the block for the actual scrapers. We ship this configuration for every client — being readable is the entire point.
Check it yourself, free
AEO Readiness Checker
Check any site for the signals that decide whether AI engines can read and recommend it.