Build a valid robots.txt file in seconds. Add rules for any user-agent, block directories, set crawl delays, and add your sitemap URL — with live syntax-highlighted preview and validation.
robots.txt is a plain text file placed at the root of your website that tells search engine crawlers and other bots which pages or sections they are allowed or not allowed to crawl. It follows the Robots Exclusion Standard — a protocol all major search engines respect by convention.
When a bot visits your site, it first checks yourdomain.com/robots.txt before crawling anything else. If the file contains rules that apply to that bot, it will follow them. If no file exists, bots treat the entire site as open for crawling.
* for all bots.Disallow: /admin/ blocks everything under /admin/.Disallow stops Google from crawling it, but Google can still index a URL it has never visited if it finds links to it on other pages. To prevent a page from appearing in search results entirely, use a noindex meta tag or X-Robots-Tag HTTP header on the page itself. robots.txt and noindex serve different purposes and are often used together.Disallow: / for their user-agents will stop them from training on your content. However, blocking AI crawlers has no impact on your Google rankings and doesn't prevent use of content that was already crawled before you added the rules. Many publishers choose to block them as a matter of principle or licensing.User-agent: * section with broad rules, and then separate sections for specific bots like Googlebot or Bingbot with different rules. When a bot reads the file, it follows the most specific section that applies to it, then falls back to * rules. Most bots stop at the first matching group and ignore other groups.