User-agent group 1
Rules
Disallow
Disallow
Disallow
Disallow
Robots.txt Generator
Create a valid robots.txt file for search engines, AI crawlers, and website directories.
- 100% private
- Instant
- Free
Quick presets
User-agent groups
Sitemap URLs
Sitemap lines appear at the bottom of the generated file.
Live preview
User-agent:: * Disallow: /admin/ Disallow: /api/ Disallow: /private/ Disallow: /tmp/
URL path tester
Enter a path to see whether each user-agent group would allow or block it with your current rules.
- *Blocked
How to use this tool
1
Choose a preset
Start from Allow All, Block All, Block AI Crawlers, Standard, WordPress, or E-commerce.
2
Adjust crawler rules
Add user-agent groups, Allow and Disallow paths, crawl-delay, and sitemap URLs.
3
Copy or download your file
Use the live preview, then copy to clipboard or download robots.txt for your site root.
Why this matters
robots.txt guides crawlers toward public pages and away from admin, checkout, and staging paths.
A mistaken Disallow: / can hide your entire site from search — presets and warnings reduce that risk.
Listing your sitemap URL helps search engines discover new and updated content faster.
Blocking AI crawlers (GPTBot, Google-Extended, and similar) is a growing choice for brands protecting training use of their content.
Clear crawl rules support technical SEO, privacy, and crawl-budget control without server changes.
Need technical SEO support?
Use this free tool to create a clean robots.txt file, or work with Neurixmedia to improve crawl control, indexing, and technical SEO.
Frequently asked questions
Yes. All Neurixmedia website utility tools are free to use with no signup required for core functionality.