Where should robots.txt be hosted?

Always place it at the root of your domain (https://example.com/robots.txt). Crawlers ignore robots.txt files placed in subdirectories.

Does Disallow in robots.txt remove a page from Google?

Not necessarily—if external sites link to a disallowed URL, Google may still index the URL without crawling its content. Use a noindex meta tag to deindex pages.

Can I block AI training crawlers like GPTBot?

Yes, you can add dedicated User-agent blocks such as GPTBot or CCBot with Disallow: / while leaving Googlebot open.

Should I disallow CSS and JS folders?

No. Modern search engines need access to CSS and JavaScript assets to render and evaluate your pages accurately.

SEO Tools100% Client-Side4.8 (482 ratings)

Robots.txt Generator

Made for site owners and DevOps engineers who need error-free Allow/Disallow rules and sitemap pointers without syntax mistakes.

User-agent

Allow (comma-separated)

Disallow (comma-separated)

Sitemap URL

Crawl-delay (optional seconds)

Generated robots.txt

User-agent: *
Allow: /
Disallow: /api/
Disallow: /login/
Disallow: /register/

Sitemap: https://toolcix.com/sitemap.xml

How to Use Robots.txt Generator

1Choose a preset (Standard Open Site, Block Admin Paths, or Custom User-Agent Rules).
2Add Allow or Disallow path patterns for specific bots or wildcard (*).
3Enter your full XML sitemap URL (e.g., https://yourdomain.com/sitemap.xml).
4Copy or download the generated robots.txt file to your site root.

Key Features

One-click presets including Next.js/WordPress/AI-crawler templates
Dynamic multi-rule builder for User-agent, Allow, and Disallow
Automatic Sitemap directive formatting
Instant robots.txt file download

Controlling Crawl Budget Without Accidentally Deindexing Your Site

A single misplaced slash in robots.txt can block search engines from crawling your entire domain. Conversely, leaving internal search endpoints, staging folders, or cart parameters wide open wastes crawl budget on low-value URLs.

ToolCix's Robots.txt Generator enforces clean line breaks, valid User-agent blocks, and unambiguous Allow/Disallow paths so your crawl instructions match the Robots Exclusion Standard.

Pairing Robots.txt With Your XML Sitemap

Including a Sitemap directive at the bottom of robots.txt gives every compliant crawler an immediate map to your canonical URLs. Combine this tool with our Sitemap Helper and Website SEO Analyzer for a rock-solid technical foundation.