Robots.txt Generator

Create a correct robots.txt file online — choose which crawlers can access which paths and download the result. Free, and generated entirely in your browser. robots.txt is the first file every crawler requests, and small syntax mistakes have outsized consequences: a stray slash can deindex a whole site, and a malformed rule is silently ignored. The generator produces clean directives for the common cases — blocking admin paths, pointing crawlers at your sitemap, and setting per-bot rules for AI crawlers like GPTBot. The table covers what each directive actually does.

robots.txt directives

DirectiveWhat it does
User-agentWhich crawler the following rules apply to (* = all)
DisallowPath prefix the crawler must not fetch
AllowException within a disallowed path
SitemapAbsolute URL of your sitemap.xml
Crawl-delaySeconds between fetches (Bing/Yandex; ignored by Google)

When to use it

  1. Block crawlers from /admin, /cart, and other pages that should never rank.
  2. Add your sitemap location so crawlers find it without Search Console.
  3. Set different rules for specific bots, including AI crawlers like GPTBot.
  4. Replace a copy-pasted robots.txt of unknown provenance with a clean one.

How to Use

  1. Choose a preset (Allow All, Block All, or common CMS presets) or start from scratch.
  2. Add rules for specific bots by selecting the User-agent and setting Allow/Disallow paths.
  3. Add your sitemap URL in the Sitemap field.
  4. Preview the generated robots.txt in the output panel.
  5. Copy or download the file and place it at the root of your website.

Features

  1. Generate robots.txt files with Allow and Disallow rules
  2. Preset configurations for common use cases (Allow All, Block All, WordPress, etc.)
  3. Support for multiple User-agent directives
  4. Sitemap URL field for linking your XML sitemap
  5. Crawl-delay setting for compatible crawlers
  6. Live preview and one-click download

FAQs

What is a robots.txt file?

A robots.txt file tells search engine crawlers which pages or sections of your site they can or cannot visit. It's placed at the root of your website.

Do all search engines respect robots.txt?

Most legitimate search engines (Google, Bing, etc.) respect robots.txt directives. However, malicious bots may ignore these rules. Robots.txt is not a security mechanism.

What does 'Allow' vs 'Disallow' mean?

Disallow tells bots not to crawl a specified path. Allow explicitly permits crawling of a path, which is useful for overriding a broader Disallow rule.

Should I include a sitemap in robots.txt?

Yes, it's good practice to include a Sitemap directive pointing to your XML sitemap. This helps search engines find all your important pages.

What is crawl delay?

Crawl-delay tells bots to wait a specified number of seconds between requests. Google ignores this directive but Bing and some other crawlers respect it.

Technical Details

Robots.txt file generation adheres to the Robots Exclusion Protocol, creating directives that control search engine crawler behavior. The tool constructs User-agent rules specifying which bots (Googlebot, Bingbot, or wildcard *) are targeted, followed by Allow/Disallow paths using pattern matching with wildcards (* for any sequence, $ for end-of-URL). Crawl-delay directives are added for rate limiting, and Sitemap declarations point crawlers to your XML sitemap location. The generator validates syntax in real-time, warns about conflicting rules (more specific paths override general ones), and prevents common mistakes like disallowing critical resources (CSS/JS files needed for Google's mobile-first indexing). Generated files include comments explaining each directive for future maintenance.

From the Blog

  • A Complete Guide to SEO Tools for Website Owners
  • Free Meta Tag Generator for SEO — Open Graph & Twitter Cards

SEO Tools

SEO Tools

  • Keyword Density Checker
  • SERP Snippet Checker
  • Hreflang Generator
  • Meta Tags Analyzer
  • Open Graph Checker
  • ads.txt Generator
  • llms.txt Generator
  • Redirect Chain Planner