Robots.txt Generator
A robots.txt file tells search engine crawlers which parts of your site they may crawl. Build one with simple rules, add your sitemap, and download a file ready to upload to the root of your domain.
Loading the tool…
How to use the Robots.txt Generator
- Choose to allow everything or start from the WordPress preset.
- Add rules for all crawlers or for a specific one, such as Googlebot.
- Add the full URL of your XML sitemap.
- Download robots.txt and upload it so it opens at yourdomain.com/robots.txt.
Questions
Does Disallow remove a page from Google?
No. Disallow stops crawling, not indexing. A blocked page can still appear if other sites link to it. Use a noindex tag on the page to keep it out of results.
Can a mistake in robots.txt hurt my site?
Yes. "Disallow: /" for all crawlers blocks your whole site. The generator warns you if your rules would do that.
Related SEO tools
XML Sitemap Generator
Turn a list of URLs into a valid XML sitemap.
Free · runs in your browser
Redirect & HTTP Header Checker
Follow every redirect and see the status code and headers of each hop.
Free · daily limit
Hreflang & Canonical Tag Checker
Check a page's canonical URL and hreflang tags for errors.
Free · daily limit
Want an expert to fix what you find?
Our SEO team can handle the technical fixes, content and links for you.