Create a correct robots.txt file online — choose which crawlers can access which paths and download the result. Free, and generated entirely in your browser. robots.txt is the first file every crawler requests, and small syntax mistakes have outsized consequences: a stray slash can deindex a whole site, and a malformed rule is silently ignored. The generator produces clean directives for the common cases — blocking admin paths, pointing crawlers at your sitemap, and setting per-bot rules for AI crawlers like GPTBot. The table covers what each directive actually does.
| Directive | What it does |
|---|---|
| User-agent | Which crawler the following rules apply to (* = all) |
| Disallow | Path prefix the crawler must not fetch |
| Allow | Exception within a disallowed path |
| Sitemap | Absolute URL of your sitemap.xml |
| Crawl-delay | Seconds between fetches (Bing/Yandex; ignored by Google) |
A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot visit. It's placed at the root of your website.
Most legitimate search engines (Google, Bing, etc.) respect robots.txt directives. However, malicious bots may ignore these rules. Robots.txt is not a security mechanism.
Disallow tells bots not to crawl a specified path. Allow explicitly permits crawling of a path, which is useful for overriding a broader Disallow rule.
Yes, it's good practice to include a Sitemap directive pointing to your XML sitemap. This helps search engines find all your important pages.
Crawl-delay tells bots to wait a specified number of seconds between requests. Google ignores this directive but Bing and some other crawlers respect it.
Robots.txt file generation adheres to the Robots Exclusion Protocol, creating directives that control search engine crawler behavior. The tool constructs User-agent rules specifying which bots (Googlebot, Bingbot, or wildcard *) are targeted, followed by Allow/Disallow paths using pattern matching with wildcards (* for any sequence, $ for end-of-URL). Crawl-delay directives are added for rate limiting, and Sitemap declarations point crawlers to your XML sitemap location. The generator validates syntax in real-time, warns about conflicting rules (more specific paths override general ones), and prevents common mistakes like disallowing critical resources (CSS/JS files needed for Google's mobile-first indexing). Generated files include comments explaining each directive for future maintenance.