Robots.txt generator.

It tells crawlers which URLs of a host they may fetch. It lives at /robots.txt on each protocol, host and port, so https://shop.example.com needs its own. A missing file (a 404) means everything may be crawled. It is a request, not a lock: polite crawlers obey it and anything else can ignore it.

No. Disallow stops crawling, not indexing: a blocked URL can still show in results, without a description, when other pages link to it. To keep a page out, let it be crawled and send a noindex robots meta tag or X-Robots-Tag header. Google stopped reading noindex inside robots.txt in 2019.

Name their tokens in a group with Disallow: /. GPTBot, ClaudeBot, CCBot, Bytespider and Meta-ExternalAgent crawl for training. Google-Extended and Applebot-Extended are control tokens rather than crawlers: Googlebot and Applebot still crawl, but Google and Apple won't use the pages for model training, and Search is unaffected. OAI-SearchBot, Claude-SearchBot and PerplexityBot feed AI search answers, so blocking them drops you from those citations.

Google ignores it, and Yandex stopped reading it in 2018; Bing still does. To slow Googlebot, answer with 503 or 429 for a while and it backs off on its own; Search Console's crawl rate setting was retired in January 2024.

Google reads the first 500 KiB and ignores the rest, so a file that long loses its last rules. Keep it short with wildcards rather than a line per URL, and save it as UTF-8 plain text.

As Google's open-source parser does: the crawler follows the group naming its most specific token (Googlebot-Image falls back to Googlebot, then to *), groups naming the same token are merged, and among the matching rules the longest path wins, with Allow winning a tie. Paths are compared after percent-encoding is normalised, and /robots.txt itself is always allowed.

Choose Paste a file and paste it. Each line Google would skip or misread gets a note (misspelled fields, a missing colon, rules before any User-agent, noindex, a relative sitemap), the tester runs on your exact lines, and Edit in the form turns it into groups you can change.

It is the same rules as an app/robots.ts file returning MetadataRoute.Robots. Next.js serves it at /robots.txt, so the rules can live in code beside the routes they describe.

No. The file is written and tested in your browser; the page makes no request with your rules or the URLs you test.

No. The Robots.txt generator is free and opens without a sign-in, an account or an install.

12 training crawlers (GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent and more), 3 control tokens (Google-Extended, Applebot-Extended, YandexAdditional), 4 AI search crawlers and 5 fetchers that act for a user, checked against their operators' documentation on 2026-10-24. Any other token can be typed in.

Yes. Googlebot-Image, Googlebot-News, Googlebot-Video and Google-InspectionTool fall back to a Googlebot group when none names them, Storebot-Google and GoogleOther go straight to *, and AdsBot-Google skips the * group, so only a group that names it can block it.

Misspelled fields Google still reads (disalow, useragent), a missing colon, rules before any User-agent line, noindex and other fields Google ignores, a sitemap given as a path, a path that doesn't start with / or *, crawl-delay, and a file past 500 KiB.

Yes. Paste the header as a crawler sends it, such as Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html), and the known token inside it is the one matched.

Closest to the Robots.txt generator: Sitemap generator, Meta tag generator, User agent parser, HTTP status codes. Each opens on its own page, free and in the browser.

The Robots.txt generator is one of 14 in its group. The group's page, /tools/web, lists them all.

The tool is free. Its interface is built from shadcn.io's own components (Button, Input, Slider and the rest of the registry under /ui), which come in React, Vue, Svelte and Solid. Pro adds the blocks and templates for the product around it.