Robots.txt Generator
Create search-engine-optimized robots.txt files offline instantly. Set crawl-delays, manage custom bot exception rules, and include sitemaps.
Robots.txt Settings
Default Crawl Access
Default Behavior for Bots
Global Crawl-Delay
Crawl-Delay Timeout
Sitemap Link
Add Directory Rule
Generated robots.txt File
What is a Robots.txt File?
A **robots.txt file** is a plain text document stored in the root directory of a website (e.g. `yourdomain.com/robots.txt`). It is part of the **Robots Exclusion Protocol (REP)**, a set of web standards that regulate how search engine web crawlers (bots) index and discover pages on your website.
By defining specific instruction sets, you tell search engine robots which directories to avoid crawling (like admin pages, temp directories, or search query URLs), helping you preserve server bandwidth and crawl budget resources for your most important landing pages.
Frequently Asked Questions (FAQ)
Does Robots.txt prevent a page from appearing in search results?
No! This is a common misconception. A robots.txt file restricts **crawling**, not indexing. If other pages on the internet link to your disallowed URL, Google may still index and show the URL in search results without reading its content. To fully prevent a page from being indexed, use a `` tag inside the HTML header.
What is a Crawl-Delay directive?
The `Crawl-delay` directive tells search crawlers to wait a specified number of seconds (e.g. 5 or 10) between each page load request. This is useful for small or shared servers that experience high CPU loads when multiple bots scan pages simultaneously. Note that Googlebot ignores crawl-delay directives (you must configure Googlebot rates inside Google Search Console).
Where should the robots.txt file be uploaded?
The file must be placed in the **root directory** of your website host (e.g. `/public_html/robots.txt` or `/var/www/html/robots.txt`). Crawlers only search for it at the exact root level (e.g. `https://example.com/robots.txt`). Placing it in a subfolder (like `https://example.com/assets/robots.txt`) will cause search engines to ignore it entirely.