Robots.txt Generator

Create robots.txt files to control search engine crawling

Website URL

User Agents

All Bots (*)

Googlebot

Bingbot

Slurp (Yahoo)

Facebookexternalhit

Twitterbot

Disallow Rules

Allow Rules

Sitemap URL (Optional)

Crawl Delay (seconds) Optional: Delay between requests (0 = no delay)