Skip to content
ToolBoxGenie

Robots.txt Generator

Developer Tools · Added 18 August 2026

Build a robots.txt group by group — user-agents, disallowed paths, allowed exceptions, sitemap lines — and get it checked as you type. The linter is the useful half: it catches the duplicate groups crawlers silently ignore, the paths that will never match, and the disallow rules that would remove a site from search or from ad serving entirely.

Replaces the rules below. Sitemap and host lines are kept.

Rule groups

Within a group the longest matching path wins, not the first one — so an Allow of /admin/public/ beats a Disallow of /admin/.

Must be absolute. This is the one directive that helps discovery rather than restricting it.

A Yandex extension. Google and Bing ignore it.

robots.txt

User-agent: *
Disallow: /admin/
Disallow: /login
Disallow: /cart
Disallow: /checkout
Disallow: /search
Disallow: /*?s=
Disallow: /*?q=

Sitemap: https://example.com/sitemap.xml

Save this as robots.txt in the root of your domain — it is only read from https://example.com/robots.txt, never from a subdirectory, and each subdomain needs its own. Serve it as text/plain.

How to use the robots.txt generator

  1. 1Pick a preset close to what you need, or start from the default group.
  2. 2Set the user-agent for each group. Use * for the catch-all and a specific name for exceptions.
  3. 3List disallowed paths one per line, and any allowed exceptions to them.
  4. 4Add your sitemap URL — it must be absolute, and it is the one directive that helps discovery.
  5. 5Read the checks below the output, then save the file as robots.txt in your domain root.

Examples

A typical CMS

Input
Block /admin/, /cart, /checkout and internal search parameters
Result
One User-agent: * group with five Disallow lines and a Sitemap line

Allowing an exception inside a blocked folder

Input
Disallow /admin/ but Allow /admin/public/
Result
Both lines are emitted — the longer path wins, so /admin/public/ stays crawlable

Opting out of AI training crawlers

Input
The AI opt-out preset
Result
Separate groups blocking GPTBot, CCBot, Google-Extended, ClaudeBot and others, while * stays open

About the robots.txt generator

What the file is actually for

robots.txt manages crawl budget, not visibility. Its best use is keeping crawlers away from URLs that are infinite, expensive or worthless to index: faceted search parameters, session URLs, internal search results, calendar pages stretching to the year 3000. On a large site, that is real money in server load and real benefit in getting the pages you care about crawled more often.

Its worst use is trying to hide something. A robots.txt is public, and listing a path in it advertises exactly where to look. Anything genuinely sensitive belongs behind authentication, not behind a Disallow line that acts as a signpost.

The rules of the format

RFC 9309 standardised what had been common practice since 1994. The essentials: rules are grouped under one or more User-agent lines; paths are site-relative and case-sensitive; * matches any sequence of characters and $ anchors the end of a URL; and an empty Disallow means everything is allowed.

Crawl-delay is not part of the standard. Google ignores it entirely and expects crawl rate to be managed in Search Console; Bing and Yandex do honour it. Host is a Yandex extension. Sitemap is universally supported and is the only line in the file that helps a crawler find more rather than less.

Mistakes that cost traffic

The expensive one is a staging robots.txt reaching production. Disallow: / on a live site stops crawling of everything, and because the pages cannot be fetched, existing entries decay out of the index over weeks. The recovery is slow even after the file is fixed.

Two subtler ones. Blocking CSS and JavaScript directories, which was once standard advice, now prevents Google from rendering the page and judging it mobile-friendly. And blocking a URL you have also asked to be de-indexed, which guarantees the noindex is never seen — the page stays in the index precisely because you blocked the instruction to remove it.

Frequently asked questions

Does Disallow keep a page out of Google?
No, and this is the most consequential misunderstanding about the file. Disallow stops crawling, not indexing. A blocked URL can still appear in results — without a snippet, sourced from links pointing at it — because Google knows the page exists but has not been allowed to look at it. To keep a page out of the index you need a noindex meta tag or an X-Robots-Tag header, which Google can only see if the page is crawlable. Blocking a page in robots.txt actively prevents the noindex from working.
Which rule wins when two match?
The most specific one, measured by path length — not the first, and not the last. Within the group that applies, the longest matching path wins, so Allow: /admin/public/ beats Disallow: /admin/. Where an Allow and a Disallow of the same length both match, Google resolves it in favour of Allow.
Why does a second group for the same user-agent get ignored?
Because a crawler reads exactly one group: the single most specific one matching its own name. Two User-agent: Googlebot groups mean Googlebot obeys one and never sees the other, and it is usually the wrong one. Merge the rules into a single group. The linter flags this because it fails silently.
Does a named group inherit the wildcard group's rules?
No, and this catches people out. If there is a Googlebot group, Googlebot uses it and ignores User-agent: * entirely — including any Disallow lines you assumed were global. Every rule that should apply to a named crawler has to be repeated inside its group.
Does blocking AI crawlers actually work?
Only for the ones that publish a token and choose to honour it — GPTBot, ClaudeBot, Google-Extended, CCBot and a handful of others do. robots.txt is a voluntary convention with no enforcement, so a crawler that ignores it is not breaking anything technical. If content genuinely must not be fetched, authentication is the only reliable answer.
Where does the file go, and does it apply to subdomains?
The root of each host: https://example.com/robots.txt. It is never read from a subdirectory, and it does not cover subdomains — blog.example.com needs its own file. It should be served as text/plain, and it applies per protocol too, so the HTTP and HTTPS versions are technically separate documents.

Related tools