Robots.txt Generator
Create custom robots.txt files to manage search engine crawler permissions, crawl delay, and sitemap directives.
Robots.txt Settings
How to Use the Robots.txt Generator
- Choose which crawler the rules apply to: all robots (*) or a specific bot such as Googlebot or Bingbot.
- Optionally set a crawl delay for crawlers that support it.
- Enter the full URL of your XML sitemap so crawlers can discover it.
- Add a Disallow rule for each folder or path crawlers should skip, such as /admin/, using "Add Rule".
- Copy or download the file and upload it to the root of your domain as /robots.txt.
Robots.txt Generator Features
- Quick rules builder for all major search engine bots
- Sitemap path integration
- Instant copy and robots.txt download
Frequently Asked Questions
Does robots.txt stop a page from appearing in Google?
No. Robots.txt controls crawling, not indexing. A blocked URL can still be indexed without its content if other sites link to it. To keep a page out of search results, use a noindex meta tag or X-Robots-Tag header, and don't block that page in robots.txt, otherwise crawlers can't see the noindex.
Does Google support the Crawl-delay directive?
Googlebot ignores Crawl-delay and adjusts its crawl rate automatically based on how your server responds. Bing and some other crawlers do respect it, so it can still be useful if a specific bot is overloading your server.
Where does the robots.txt file need to be?
It must be at the root of the host it applies to, for example https://example.com/robots.txt. Each subdomain and protocol needs its own file, so https://blog.example.com needs a separate robots.txt.
Should I block CSS and JavaScript files?
No. Search engines need your CSS and JavaScript to render pages the way visitors see them. Blocking these files can cause pages to be misunderstood and ranked lower.