Robots.txt Generator

How to use

1. Start with the * group, which applies to every crawler.
2. Add paths to disallow, one per line, each starting with /.
3. Add your sitemap as a full URL.
4. Use the path checker to confirm a URL is treated the way you expect, then copy or download the file to your site root.

Features

• Groups per crawler, with allow, disallow and crawl delay.
• A path checker that applies the real rule: longest match wins, and Allow beats Disallow at equal length.
• Warns before you block your whole site, which is the mistake that costs the most.
• Points out that a path listed here is published, not hidden.
• Runs in your browser; the file is yours to download.

Frequently asked questions

Will robots.txt hide a page?

No. It asks well-behaved crawlers not to fetch a path, and it is a public file at a predictable address — so listing /admin tells everyone, including the people you least want to tell, exactly where /admin is. It is the first file an attacker reads. Use authentication for anything that must not be reached, and noindex for anything that must not be listed.

What is the difference between robots.txt and noindex?

robots.txt stops a crawler fetching the page; noindex stops it appearing in results. They are not interchangeable, and combining them backfires: a page blocked in robots.txt can still be listed from links elsewhere, and because the crawler was never allowed to fetch it, it never sees the noindex telling it not to. If you want a page out of search, allow crawling and use noindex.

Does Disallow: / block everything?

Yes — it removes your whole site from search for any crawler that obeys it. It is one character away from Disallow: /admin, which is why this tool says so plainly rather than quietly generating it.

Why is my Crawl-delay ignored?

Google stopped supporting it; the crawl rate is set in Search Console instead. Bing and Yandex still honour it, so the line is worth keeping if those matter to you.

Related tools