Control how search engines index a page. Choose index or noindex, follow or nofollow, and options like noarchive, nosnippet and image preview size, and get the robots meta tag, the X-Robots-Tag header for PDFs and other files, and server config — with warnings for combinations that don't work.
| Directive | Effect |
|---|---|
| noindex | Exclude from results |
| nofollow | Don't follow links |
| nosnippet | No text snippet |
| max-image-preview:large | Allow large images |
| noarchive | No cached copy |
robots.txt stops crawling; noindex removes a page from results — and only works if the page can be crawled.
Send the X-Robots-Tag: noindex header for that file.
No — that's the default.
Allows large image previews, which can help visibility in Google Discover.
Yes — choose Googlebot.
The directives are combined into one comma-separated value: - index or noindex, and follow or nofollow. - noarchive, nosnippet, noimageindex and notranslate. - max-snippet:n, max-image-preview:none, standard or large, and max-video-preview:n. - unavailable_after, written as an ISO date-time. The value can target all robots or a named crawler such as googlebot or bingbot. It's emitted as a meta tag for HTML, an X-Robots-Tag header for any file type (the crawler name is prefixed when specific), and Nginx and Apache configuration lines. Conflicts are flagged: snippet settings do nothing under noindex, nosnippet overrides max-snippet, index-follow alone is the default, and a noindexed URL must not be blocked in robots.txt or crawlers never see the directive.