AI Crawler
Checker
Enter your URL. AISearch Global checks for a blanket robots.txt block and a page-level noindex tag — the two most common ways a site accidentally blocks crawling and indexing for every bot, AI or otherwise.
AISearch Global fetches your live site — no signup required
AISearch Global AI Crawler Checker · aisearch.global
We could not check this site. Please check the URL and try again.
yourdomain.com/robots.txt) to allow the specific user-agents you want reading you — GPTBot, ClaudeBot, PerplexityBot, Google-Extended and similar. A blocked crawler here means that AI system may never see your content at all.Need help deciding which findings matter?
Crawler access is one signal. Get a checked, prioritised DIY audit with AISearch Global's $1,500 Founder Audit — dated AI-search observations plus builder-specific fix instructions.
Built by AISearch Global · Sydney, Australia
About this tool
The AISearch Global AI Crawler Checker looks at the two most common ways a site
accidentally blocks AI systems from reading it: a blanket Disallow: / rule in robots.txt
under User-agent: *, and a page-level noindex directive in the meta robots tag.
Either one can silently remove your business from consideration by ChatGPT, Perplexity, Gemini, and
other answer engines, even if the rest of your AEO signals are strong.
These are two different controls, worth keeping separate: crawling access (robots.txt)
governs whether a bot can fetch the page at all, and indexing (the meta robots
noindex tag) governs whether a page that was crawled gets kept and shown. A third control —
AI training-data opt-outs such as Google's Google-Extended — is different
again: it governs whether your content can be used to train or ground a model, separately from whether
Google's own Search crawler can still crawl and index the page. Google is explicit that Googlebot's
Search-crawling role and Google-Extended's training/grounding role are two different switches, not one
"AI access" toggle. This tool checks crawling access and indexing only; it does not check for
training-data opt-out directives.
For a full technical AEO review beyond crawler access, see the AISearch Global Founder Audit.
Frequently asked questions
What does the AI Crawler Checker actually check?
It fetches your site's robots.txt and checks whether it disallows all crawlers under a User-agent: * block, and it checks the page's meta robots tag for a noindex directive. These are the two most common ways a site accidentally blocks AI crawlers.
Why would AI crawlers be blocked?
Many sites use a blanket Disallow: / rule in robots.txt intended for staging environments, or a noindex tag left over from development, without realising it blocks every crawler from reading the page, including the ones AI systems use to source and cite answers.
Is this the same as an AI training-data opt-out like Google-Extended?
No. This tool checks crawling access (robots.txt) and indexing (the meta robots noindex tag) only. Google-Extended and similar directives are a separate control: they govern whether your content can be used to train or ground an AI model, independent of whether Google's own Search crawler can still crawl and index the page. This tool does not check for those training-data opt-out lines.
If AI crawlers are blocked, can I fix it myself?
Yes. Remove the blanket disallow rule from robots.txt or the noindex meta tag from the page, then re-run this check to confirm. If you're unsure which crawlers to allow, AISearch Global's Founder Audit reviews this as part of a full technical AEO check.