Robots.txt Generator
Create robots.txt files to manage search crawler permissions and declare sitemap locations.
The Engineering of Crawler Governance: RFC 9309, Pattern Matching & Crawl Budget Control
Before any legitimate search engine crawler—such as Googlebot, Bingbot, or DuckDuckBot—requests a single HTML document, CSS stylesheet, or image file from your web server, it issues an HTTP GET request to the domain root for a special plain-text file: /robots.txt.
Originally conceived in 1994 by Martijn Koster and officially formalized by the Internet Engineering Task Force (IETF) in 2022 as RFC 9309 (Robots Exclusion Protocol), the robots.txt file serves as the fundamental access control gate between webmasters and automated automated crawlers. A properly constructed robots.txt optimizes your server's Crawl Budget, shields private administrative routes, and prevents server overload from aggressive web scrapers.
Robots.txt Directive Syntax & Crawler Behavior Reference Table
The comparative reference table below breaks down the primary RFC 9309 directives, pattern matching logic, and algorithmic consequences across search crawlers:
| Directive Syntax Example | Pattern Matching Mechanism | RFC 9309 Crawler Action | Common Production Use Case | Critical Risk or Pitfall |
|---|---|---|---|---|
Disallow: / |
Root prefix match (all paths) | Complete Crawl Block | Staging, development, or pre-launch private portals | Accidental production deployment de-indexes entire website |
Disallow: /admin/ |
Directory prefix match | Blocks requests starting with /admin/ |
Securing backend panels, internal dashboards, login paths | Exposes internal URL paths publicly in plain text |
Allow: /public/ |
Explicit path override | Permits crawl within an otherwise disallowed directory | Whitelisting public assets inside a restricted folder | Longest-match rule applies if conflicting with Disallow |
Disallow: /*.pdf$ |
Wildcard regex pattern (end of URI) | Blocks all URLs ending with .pdf extension |
Preserving crawl budget from heavy binary document files | Prevents indexing of valuable customer guides or whitepapers |
Disallow: /*?*sort= |
Query parameter pattern | Blocks infinite faceted e-commerce filter permutations | E-commerce category filter and query deduplication | Accidental blockage of canonical product URLs |
Sitemap: https://... |
Absolute URI declaration | Registers authoritative XML sitemap location | Directing crawlers to the site's complete content roadmap | Relative URLs are ignored; must be absolute HTTPS |
The Critical Distinction: Crawling Block vs. Indexing Removal
One of the most dangerous and common misconceptions in web development is believing that adding a page to robots.txt with Disallow removes it from Google search results. It does not.
RFC 9309 dictates crawling permissions, not indexing rules:
- When a URL is Disallowed: Googlebot will never request the page from your server. It cannot read the HTML, cannot execute on-page scripts, and cannot see on-page tags.
- Why It Can Still Appear in Search: If external websites (or internal pages) link to that disallowed URL, Google's indexing algorithm may still display the URL in search results as a naked link snippet with the message: "A description for this result is not available because of this site's robots.txt".
- The Correct Removal Protocol: To completely eliminate a page from search results, you must ALLOW the crawler to access the page and include a
<meta name="robots" content="noindex">tag in the HTML<head>or serve anX-Robots-Tag: noindexHTTP response header. If robots.txt blocks the crawler, the spider can never read thenoindextag!
Prefix Matching Rules and Trailing Slashes
Pattern matching in robots.txt operates on simple string prefix evaluation. Omitting a trailing slash can trigger catastrophic unintended crawl blocks:
Disallow: /order
# The rule above blocks:
# https://example.com/order/
# https://example.com/order-history
# https://example.com/orderly-guide-to-seo (UNINTENDED!)
# SAFE: Strict directory matching
Disallow: /order/
Managing Crawl Budget for Large Scale Applications
Search engines assign every domain a finite Crawl Budget—the cumulative number of simultaneous connections and pages a crawler will fetch before departing to other domains. If your application generates thousands of faceted search combinations, infinite calendar loops, or internal search parameter strings, spiders may waste their entire crawl allowance on low-value pages while fresh editorial articles and products remain unindexed.
By implementing strategic Disallow patterns for internal search routes (e.g., Disallow: /search?*) and temporary cart sessions, you focus search engine bandwidth entirely on high-value canonical content.
Frequently Asked Questions
Where must the robots.txt file be uploaded on my server?
The robots.txt file must reside in the absolute root directory of your domain (e.g., https://yourdomain.com/robots.txt). Placing it in subdirectories (such as /assets/robots.txt) will cause search engines to ignore it entirely.
Does blocking a page in robots.txt prevent it from showing on Google?
No. Blocking a page in robots.txt prevents the crawler from requesting the page content, but Google can still index the URL if external links point to it. To guarantee de-indexing, you must allow crawling and apply a noindex meta tag.
What does `User-agent: *` mean in a robots.txt file?
The asterisk (*) is a universal wildcard that applies the declared rules to all compliant search engine bots and automated crawlers (Googlebot, Bingbot, DuckDuckBot, etc.) unless an explicit bot rule overrides it.
Is robots.txt case-sensitive?
Yes. URL path directives are strictly case-sensitive under RFC 9309. For example, Disallow: /private/ will not block /Private/ or /PRIVATE/.
Can I declare multiple sitemaps in one robots.txt file?
Yes. You can declare multiple Sitemap: lines (e.g., one for regular pages, one for a blog, and one for image sitemaps) anywhere within the file, or reference a single master Sitemap Index file.
Are malicious scrapers and bad bots blocked by robots.txt?
No. robots.txt is a voluntary protocol honored by reputable search engines. Malicious bots, email scrapers, and content scrapers routinely ignore robots.txt directives. Preventing unwanted scrapers requires server-level firewall rules (WAF) or IP rate limiting.