Robots.txt File Generator

Generate valid robots.txt files to guide search engine crawlers and protect private directories.

Crawl Rules & Permissions

Disallowed Paths

Specific Search Engine Bots

Generated robots.txt

How to Use Robots.txt Maker

  1. Select default crawling policy (Allow All or Disallow All).
  2. Add paths to exclude from indexing (e.g. /admin/, /private/, /tmp/).
  3. Copy the generated robots.txt text or download the file directly.

Technical Specifications & How It Works

Robots.txt instructs web robots on which pages or files the crawler can or cannot request from your site.

100% Client-Side Privacy: Server paths and domain URLs are kept private in your browser.

Frequently Asked Questions

Where should robots.txt be placed on my website?

The file must always be placed at the root level of your website: `https://yourdomain.com/robots.txt`.

Can I block specific crawlers like Baidu or Yandex?

Yes. Configure individual rules for Googlebot, Bingbot, Baiduspider, and Yandex.

How do I declare my XML sitemap?

Enter your full sitemap URL in the Sitemap field, and it will be appended to the bottom of the file.

More Free Browser Tools

Developer
Sitemap Generator Generate valid standard XML sitemaps for Google Search.
Developer
SERP Preview Preview Google desktop and mobile search snippets in real time.
Developer
Meta Tags Maker Generate complete SEO, Open Graph, and Twitter Card tags.
Developer
File Details Inspect byte size, MIME type, SHA-256 hash, and hex dump.