Robots.txt Generator
Write a robots.txt from the paths you allow and block.
Write a robots.txt from the paths you allow and block. Save it at the site root yourself.
User-agent: * Allow: / Disallow: /private Sitemap: https://example.com/sitemap.xml
Calculations happen in your browser.
Worked example
User agent *, allow /, disallow /private, and a sitemap URL produces a robots.txt with those four lines. Blank lines in the path boxes are skipped.
FAQ
- Does this publish the file?
- No. It only writes the text. Save it as robots.txt at the root of your site.
- What does Disallow: /private mean?
- Paths that start with /private are asked to stay out. Crawlers may still ignore the file. It isn’t access control.
- Can I target one bot?
- Change the user agent from * to the bot’s token, such as Googlebot. This form writes a single group.