Robots.txt
Checker
Enter your URL. AISearch Global fetches robots.txt from your site's root and checks whether it exists, what it allows, and whether it's blocking everyone.
AISearch Global fetches your live site — no signup required
AISearch Global Robots.txt Checker · aisearch.global
We could not check this site. Please check the URL and try again.
yourdomain.com/robots.txt). If it's missing, blocking crawlers you want, or needs a sitemap reference, add or edit it through your CMS's file manager, your static site's source, or your server's document root.Need help deciding which findings matter?
Robots.txt is one signal. Get a checked, prioritised DIY audit with AISearch Global's $1,500 Founder Audit — dated AI-search observations plus builder-specific fix instructions.
Built by AISearch Global · Sydney, Australia
About this tool
The AISearch Global Robots.txt Checker fetches /robots.txt from a site's
root and reports whether it exists, how many directive lines apply to all crawlers under
User-agent: *, and whether a blanket Disallow: / rule is quietly blocking every
crawler, including the ones that power AI-generated answers.
The exact rule applied: this tool fetches yourdomain.com/robots.txt at the site's origin only
— the root file, not any specific page path or URL — and reads the directive lines under the
User-agent: * block only.
It does not check per-bot sections such as a separate User-agent: GPTBot block elsewhere in
the file, and it doesn't test whether any single path is disallowed, only whether the wildcard section
contains a blanket Disallow: /.
For page-level blocking as well, use the AI Crawler Checker, which also checks meta robots directives.
Frequently asked questions
What does the Robots.txt Checker look for?
It fetches robots.txt from your site's root, confirms it exists, counts the directive lines that apply under User-agent: *, and flags whether a Disallow: / rule is blocking every crawler from the entire site.
What if I don't have a robots.txt file?
No robots.txt isn't automatically a problem, crawlers will still access your site by default, but publishing one gives you explicit control and is considered good practice for both SEO and AEO.
Is a robots.txt checker the same as an AI crawler checker?
This tool checks robots.txt specifically. The AISearch Global AI Crawler Checker goes a step further, also checking page-level meta robots directives that can separately block AI systems.
Which exact path and rule does this tool test?
It fetches the robots.txt file at the site's origin only (yourdomain.com/robots.txt), not any specific page path, and reads the directive lines under the User-agent: * block only. It does not check per-bot sections such as a separate User-agent: GPTBot block elsewhere in the file.