Crawler & Robots.txt Checker

It reads the robots.txt file your website publishes and reports what that file instructs, for each crawler listed, at the one URL you submitted. That is a configuration reading: it tells you what your site asks crawlers to do.

No. robots.txt is one layer. A CDN, firewall, WAF, bot-protection rule, geo-block or login can still refuse a crawler that your robots.txt permits, and this check does not test any of those. An allowed result means your published instructions do not refuse that crawler at that path — nothing more.

Explicitly allowed means a rule in your file names that path and permits it. No applicable rule means your file simply does not address it — the crawler is not named and no catch-all rule matches the path either. Both commonly result in the page being fetched, but only one of them is something your file actually says, so this tool reports them separately rather than calling both "allowed".

Because robots.txt rules apply per path. A rule can permit your homepage and refuse /members, so a single whole-site verdict would be wrong. The result applies to the exact URL you submitted. Submit another path to check that path.

Then it reports that it could not read the file, and every crawler is marked undetermined. It does not report crawlers as allowed, because a file that could not be read is not evidence of permission.

This check cannot tell you that, and neither can your robots.txt. Crawler access is a technical configuration matter. Whether an AI assistant names a particular business in an answer depends on many things this file does not control, and no robots.txt change guarantees a citation, a mention or a recommendation.

No. It makes exactly one request, to /robots.txt, identifying itself as TendorAI Robots Checker. An earlier version of this tool sent twelve requests carrying other companies’ crawler names; that behaviour was removed because it misrepresented who was connecting to your server.