Robots.txt Generator

robots.txt tells well-behaved crawlers which parts of your site they may request. It is not a security feature — blocked URLs can still be visited and can even appear in search results if other pages link to them. To keep a page out of results, use a noindex tag instead.

Preset

Sitemap URL

Disallow paths (one per line)

Allow paths (exceptions)

Also block common AI-training crawlers (GPTBot, CCBot, Google-Extended, ClaudeBot, Applebot-Extended, PerplexityBot, Bytespider)

robots.txt

Test a path

Path to test

Runs entirely in your browser. Your input and files are not uploaded to any server.

How to use

Pick a starting preset.
List the paths to disallow or allow, one per line, and add your sitemap URL.
Test a few paths, then download robots.txt and upload it to the root of your domain.

Examples

Block an admin area

Disallow /admin/ while leaving the rest of the site crawlable.

Opt out of AI training crawlers

Add rules for user agents such as GPTBot and CCBot that publish opt-out instructions.

Privacy

Runs entirely in your browser. Your input and files are not uploaded to any server. See the privacy policy for what is stored in your browser (favorites and history only). Privacy policy

Check it yourself: open your browser’s developer tools, select the Network tab and use the tool. Your file never appears in an upload request. Once this page is saved for offline use, you can also disconnect from the internet and keep working. Ads and anonymous visit statistics, when enabled, make their own requests, but they never receive your files or text.

Frequently asked questions

Where does robots.txt go?

At the root of the host: https://example.com/robots.txt. Each subdomain needs its own file.

Does Disallow remove a page from Google?

No. It stops crawling, not indexing. Use a noindex meta tag (and allow crawling) to remove a page from results.

How does the tester decide?

It follows the Robots Exclusion Protocol (RFC 9309): the longest matching rule wins, Allow wins a tie, and * and $ wildcards are supported. It tests the group for “*”.

Related tools

All SEO Tools

XML Sitemap GeneratorPaste a list of URLs and get a valid, escaped sitemap.xml with optional last-modified dates.

Meta Tag GeneratorGenerate clean HTML meta tags for a page: title, description, canonical, robots, Open Graph and Twitter Card tags, with length guidance.

Canonical URL HelperNormalise URLs to one consistent canonical form: HTTPS, www or non-www, trailing slash, tracking parameters removed.

Favicon GeneratorGenerate a complete favicon set from an image or a letter: favicon.ico, 16–512 px PNGs, Apple touch icon, web manifest and HTML tags, in one ZIP.

Slug GeneratorTurn titles into clean, lowercase URL slugs.

Open Graph PreviewPreview a link card from your og:title, og:description and og:image, check image size and text length, and copy the Open Graph tags.