What does robots.txt control?
robots.txt is a text file at the root of a host that communicates crawler access preferences using directives such as User-agent, Allow and Disallow. It is not an access-control mechanism and does not prevent a URL from being known or indexed in every circumstance.
How to use the Robots.txt Checker
- Enter the URL, text or details requested by the tool.
- Run the check or generator.
- Review the result and apply changes only after checking them against your site and content.
SEO guidance
Use this tool as one part of an SEO workflow. Automated checks are useful for finding implementation problems quickly, but they do not replace manual review of search intent, content quality, crawlability, internal linking and the overall site architecture.
Frequently asked questions
Is this tool free?
Yes. The standalone checker page is available without an API key.
Can I check any website?
The tool is designed for publicly accessible HTTP or HTTPS pages. Private, local and reserved network addresses are blocked.
Does the result guarantee Google rankings?
No. These checks identify implementation issues and provide practical diagnostics; search rankings depend on many factors.
What does the score mean?
It is a weighted diagnostic score based on the checks included in this version. It is not a Google score.
Does robots.txt block all crawlers?
No. Different crawlers may interpret directives differently, and robots.txt is not an access-control system.