Robots.txt Generator

Create robots.txt file to control search engine crawlers

📝 Quick Templates

Configuration

User-agent

Use * for all crawlers or specify (e.g., Googlebot)

Disallow

Allow

Crawl-delay (seconds)

Sitemap URL

Generated robots.txt

User-agent: *

About Robots.txt

The robots.txt file tells search engine crawlers which pages or files they can or can't request from your site. It's placed in the root directory of your website.

Disallow: Blocks crawlers from accessing specific paths
Allow: Explicitly allows access to specific paths
Sitemap: Points crawlers to your XML sitemap