Robots.txt generator
Build a robots.txt file for your website. Control which crawlers can access which parts of your site.
What is robots.txt?
Robots.txt is a plain text file placed at the root of your website (e.g., yoursite.com/robots.txt) that instructs web crawlers which pages or directories they are and aren't allowed to access. It follows the Robots Exclusion Protocol — an informal standard followed by all major search engines and most well-behaved bots.
The file uses simple directives: User-agent specifies which bot the rule applies to (* means all), Allow permits access to a path, Disallow blocks it, and Sitemap points crawlers to your sitemap. Robots.txt is a request, not a security measure — malicious bots ignore it entirely. Never use robots.txt to hide sensitive pages; use proper authentication instead. Common use cases: blocking crawlers from /admin, /checkout, staging environments, or duplicate URL patterns with parameters. Since 2023, you can also use robots.txt to block AI training scrapers like GPTBot, CCBot, and Google-Extended.