Robots.txt Generator
robots.txt Rule Generator
Supports multi-line Allow / Disallow / Sitemap, automatically assembles standard format.
GeekFormat's online Robots.txt Generator lets you configure User-agent, Allow/Disallow rules, Sitemap declarations, and Host directives through a visual form, generating robots.txt files compliant with the Robots Exclusion Protocol in real time. The page opens with default example values—modify any field to preview results instantly, then copy with one click to deploy to your website root directory. Runs entirely locally in your browser; no configuration data is uploaded to any server.
Related
About Robots.txt and the Robots Exclusion Protocol
robots.txt is one of the most basic ways websites communicate with search engine crawlers, full name Robots Exclusion Protocol (REP). It is a plain text file placed in a website's root directory that uses simple directives to tell compliant search engine crawlers which parts of the website are allowed to be crawled and which parts are not desired to be crawled. robots.txt was first proposed by Martijn Koster in 1994; after nearly 30 years of development, it was formally standardized by the IETF as RFC 9309 in 2022.
The basic structure of robots.txt consists of one or more rule groups. Each rule group starts with one or more User-agent lines specifying the crawlers to which the rules apply (* means all crawlers); followed by several Allow and Disallow lines, respectively specifying path prefixes allowed and disallowed for crawling; non-group-level directives such as Sitemap and Host can be added at the end of the file. Each rule group is separated by blank lines. A typical robots.txt contains: User-agent declaration → Allow/Disallow path rules → (optional) more User-agent groups → Sitemap declaration → Host directive.
The User-agent field specifies the crawler name to which the rules apply. Common search engine crawler names include: Googlebot (Google Search), Bingbot (Bing Search), Baiduspider (Baidu Search), YandexBot (Yandex Search), DuckDuckBot (DuckDuckGo Search), Twitterbot (Twitter/X card fetching), facebookexternalhit (Facebook link preview), GPTBot (OpenAI GPT crawler), Bytespider (ByteDance/Toutiao Search), etc. Use User-agent: * as a wildcard to match all crawlers not separately matched.
Allow and Disallow directives use the Prefix Matching principle: as long as a URL path starts with the specified value, the rule matches. For example, Disallow: /admin blocks all paths starting with /admin such as /admin, /admin/, /admin/login.html, /administrator, etc. If you only want to exactly match the /admin/ directory, write Disallow: /admin/ (trailing slash added). According to RFC 9309 standards, when Allow and Disallow both match, the rule with the longer path wins; when lengths are equal, Allow takes precedence. This means more specific rules have higher priority.
The Sitemap directive tells search engines the location of sitemaps, helping crawlers discover and index website pages more efficiently. Sitemap lines must be placed after all rule groups (or at the end of the file), and URLs must be full absolute addresses (including http:// or https://). Multiple Sitemaps can be declared in one robots.txt, each on its own line. Large sites typically split into multiple Sitemaps (categorized by content type, update frequency), managed uniformly through a Sitemap index file.
The Host directive is a non-standard but widely used directive proposed by Yandex, used to specify a site's preferred domain (primary mirror). For example, when both example.com and www.example.com exist, Host: example.com tells search engines to prefer example.com. Note: Google has stated it does not use the Host directive; preferred domains need to be set through Google Search Console; Bing has not explicitly supported it either. The Host directive should be placed after Sitemap and can only appear once.
Correctly configuring robots.txt has important SEO significance: first, it prevents crawlers from wasting crawl budget on meaningless pages (such as search result pages, filtered pages, duplicate content pages), allowing crawlers to crawl valuable content more efficiently; second, it prevents private or low-value pages (such as backends, test pages, print pages) from being indexed; third, it proactively guides crawlers to discover new pages through Sitemaps. But note: robots.txt is a gentlemen's agreement and cannot replace real security measures; Disallowed URLs may still show URLs in search results if there are external links pointing to them (content just won't be crawled); completely blocking crawling may actually affect overall site weight assessment.
Common robots.txt errors include: ① Wrong paths (such as Disallow: admin missing leading slash, should be /admin); ② Using regular expressions (standard REP does not support regex, only Googlebot supports limited wildcards * and $); ③ Blocking JS/CSS files (which prevents Google from rendering pages); ④ Disallow: / (blocking the entire site, a fatal error that causes the site to not be indexed at all); ⑤ File encoding issues (must be UTF-8 encoding); ⑥ Placed in a subdirectory instead of root directory; ⑦ File permissions preventing access (should return 200 OK status code). It is recommended to verify after generation using Google Search Console's robots.txt testing tool or this platform's robots.txt inspection tool.
Use Cases
- Configure a basic robots.txt before launching a new site to clearly define allowed and disallowed crawl paths, guiding search engines to crawl correctly
- Update Disallow rules after a site redesign to prevent old directories from continuing to be crawled, which could affect SEO weight distribution
- Add multiple Sitemap address declarations to help search engines discover full site page structure faster, improving indexing efficiency
- Block backend admin directories (/admin), test environments, and private directories from being indexed by crawlers, reducing security risks
- Set the Host directive to specify your site's preferred domain (with or without www), reducing duplicate content issues
- Configure independent crawl rules for different search engine crawlers (Googlebot, Bingbot, Baiduspider, etc.)
- Temporarily block crawlers from accessing directories under maintenance, reopening crawl access once site updates are complete
- Generate properly formatted robots.txt files to avoid search engine parsing failures caused by handwritten formatting errors
- Configure multiple Sitemaps (e.g., article sitemap, product sitemap, image sitemap) to cover all site content types
- Frontend development demonstration of robots.txt rule configuration, teaching standard crawler protocol formatting
- Use with a robots.txt inspection tool to verify that generated rules match expectations, avoiding accidental blocking of important pages
- Quickly create a robots.txt template, download it, fine-tune it for your actual site, then deploy
How to Use
- Specify the target crawler in the User-agent input box (default * represents all search engine crawlers)
- Fill in paths allowed for crawling in the Allow area, one rule per line (e.g., /, /blog/)
- Fill in paths disallowed for crawling in the Disallow area, one rule per line (e.g., /admin, /private)
- Add XML sitemap addresses in the Sitemap area (multi-line supported), and fill in your preferred domain in the Host area
- The right preview area displays generated robots.txt content in real time; once confirmed correct, click the copy button to deploy
Features
- Independent multi-rule configuration: Separate input areas for User-agent, Allow, Disallow, Sitemap, and Host keep rules clearly categorized instead of mixed together
- Real-time preview generation: robots.txt content updates instantly when any rule is modified—no need to manually click a generate button, what you see is what you get
- Default fallback rule: Automatically outputs Allow: / when both Allow and Disallow are empty, preventing blank files that cause undefined crawler behavior
- Multi-Sitemap support: The Sitemap area supports multi-line input, with each URL output as an independent Sitemap declaration line—perfect for sites with multiple Sitemaps
- One-click copy for deployment: Results can be copied to the clipboard with one click, ready to paste directly into your website root directory for deployment
- Default example pre-filled: The page opens with default example configuration (User-agent: *, Allow: /, Disallow: /admin /private, Sitemap, Host) that beginners can reference and modify directly
- Standard format output: Strictly follows Robots Exclusion Protocol specifications—User-agent lines first, rule lines after, Sitemap/Host at the end—compatible with all search engines
- Smart path processing: Automatically trims leading/trailing whitespace from each line and filters empty lines, preventing rule failure due to extra spaces
- Split-pane responsive layout: Left-right split panes (configuration/preview) on desktop, stacked top-bottom on mobile—works great on both desktop and phones
- Monospace font preview: The preview area uses monospace font to display generated results, keeping robots.txt formatting neat and clear
- Host directive support: Configure your preferred Host domain (supported by Yandex and other search engines) to reduce duplicate content issues across domains
- Runs entirely locally in the browser: All configuration and generation operations are completed in local browser JavaScript—no data is sent to any server
- Zero-dependency instant use: Ready to use as soon as the page opens—no registration, login, or software installation required
- Linked with inspection tool: Related links lead directly to the robots.txt inspection tool, so you can verify rules immediately after generation
FAQ
What is robots.txt for? Why does a website need this file?
robots.txt is a plain text file placed in a website's root directory, used to tell search engine crawlers (such as Googlebot, Bingbot, Baiduspider, etc.) which paths can be crawled and which are forbidden. It is the implementation of the Robots Exclusion Protocol (REP), the core file for site crawl management and basic SEO setup. While not strictly required, nearly all legitimate websites configure robots.txt to guide crawler behavior.
Where should the robots.txt file be placed?
robots.txt must be placed in the website's root directory and accessible directly at http://yourdomain/robots.txt, for example https://example.com/robots.txt. Note that placing it in a subdirectory (such as /blog/robots.txt) is invalid—crawlers only look for this file in the root directory. The filename must be all lowercase: robots.txt, not Robots.txt or ROBOTS.TXT.
What content is generated when both Allow and Disallow are empty?
When neither Allow nor Disallow is filled in, the generator automatically outputs Allow: / as a fallback rule, meaning crawling of the entire site is allowed. This prevents blank files or incomplete robots.txt files with only User-agent lines, avoiding undefined crawler behavior due to unclear rules.
Can I configure multiple Sitemap addresses?
Yes. The Sitemap area supports multi-line input, and each URL will be output as an independent Sitemap declaration line. For example, large sites typically have multiple sitemap files (article sitemap, product sitemap, image sitemap, etc.)—you can fill in one complete Sitemap URL per line (must be a full absolute path including http:// or https://).
What does the Host directive do? Do all search engines support it?
The Host directive specifies a site's preferred domain (such as example.com or www.example.com), helping search engines identify the primary domain and reduce duplicate content issues between www and non-www versions. Currently the Host directive is mainly supported by search engines like Yandex; Google does not directly use the Host directive, instead setting preferred domains through Google Search Console. It is recommended to configure it but not rely solely on it.
What does User-agent: * mean? How do I configure rules for specific crawlers?
User-agent: * means the rules apply to all crawlers (the asterisk is a wildcard). If you need to configure independent rules for specific search engines, you can set User-agent to a specific crawler name, such as Googlebot (Google), Bingbot (Bing), Baiduspider (Baidu), Twitterbot (Twitter/X), etc. You can configure multiple User-agent groups, separated by blank lines between each group.
Can robots.txt truly protect sensitive directories from access?
No. robots.txt is only a gentlemen's agreement—compliant search engine crawlers will follow the rules, but malicious crawlers and hacker scanners can completely ignore it. Do not rely on robots.txt to protect truly sensitive content (such as backend passwords, user private data); sensitive directories should use real security measures like server-side authentication (password protection, IP whitelisting). The purpose of robots.txt is to guide compliant crawlers, not security protection.
What is the path matching rule for Disallow?
Disallow rules use prefix matching: Disallow: /admin matches all paths starting with /admin, such as /admin, /admin/, /admin/login.html, /administrator, etc. If you only want to match content under the /admin/ directory, write Disallow: /admin/ (with trailing slash). Disallow: (empty value) means no paths are disallowed (i.e., all allowed). Note that rules are case-sensitive.
How do I deploy the generated robots.txt to my website?
Click the copy button to copy the generated content to your clipboard, then create a plain text file named robots.txt on your server, paste the content, and upload the file to your website root directory (usually the web root, public_html, www, or dist directory). After deployment, verify it works by accessing https://yourdomain/robots.txt; you can also use Google Search Console's robots.txt testing tool to verify rules.
How long does it take for changes to robots.txt to take effect?
Search engine crawlers will re-fetch the robots.txt file the next time they visit your site, usually taking effect within hours to days. If you want search engines to discover the update as soon as possible, you can submit a robots.txt update request in Google Search Console or Bing Webmaster Tools. Note: Pages already indexed may remain in search results for a period even after being Disallowed; complete removal requires using the noindex tag or URL removal tool.
Which takes precedence when Allow and Disallow conflict?
According to RFC 9309 (the formal Robots Exclusion Protocol standard), when Allow and Disallow rules have the same path length, Allow takes precedence over Disallow; when path lengths differ, the rule matching the longer path takes precedence. For example, when Allow: /blog conflicts with Disallow: /blog/, accessing /blog/post.html matches Disallow (longer path /blog/ > /blog), while accessing /blog itself matches Allow. Simply put, more specific rules have higher priority.
Can I add comments in robots.txt?
Yes, lines starting with # are comment lines, and crawlers ignore content after #. Comments can be on their own line or at the end of a rule line (after #). For example: # Block admin area or Disallow: /admin # admin panel. Adding appropriate comments helps you and team members understand what each rule does.
Do Sitemap URLs need to be absolute paths?
Yes, Sitemap directives must use full absolute URLs (including protocol and domain), such as Sitemap: https://example.com/sitemap.xml. Relative paths (such as /sitemap.xml) cannot be used because crawlers may access your site from different domains (such as example.com and www.example.com), and relative paths would prevent crawlers from determining the correct address.
Will configuration data be uploaded to the server?
Not at all. All configuration and generation operations for robots.txt are completed locally in your browser via JavaScript—configuration data such as paths and domains you enter are not sent to any external server. The page can be used offline (basic functionality) once loaded; data is automatically cleared when you close the page, leaving no records.
Can the generated robots.txt be used on any website?
Yes, the generator outputs a plain text file in standard Robots Exclusion Protocol format, suitable for any web server (Nginx, Apache, IIS, Caddy, etc.) and any site-building platform (WordPress, Shopify, Next.js, Django, Rails, etc.). Simply deploy it to your website root directory and ensure it is publicly accessible.
Troubleshooting
Generated file returns 404 error when accessed?
Confirm that the robots.txt file has been uploaded to the website root directory (not a subdirectory), the filename is all lowercase as robots.txt, and file permissions are set to publicly readable (usually 644 permissions are sufficient). After uploading, visit https://yourdomain/robots.txt directly in a browser to confirm you can see the content. Root directory locations vary by server: Nginx/Apache is usually /var/www/html/ or /usr/share/nginx/html/, Vercel/Netlify is usually the public/ directory.
Disallow rules not working, pages still getting indexed?
Possible reasons: ①Crawlers haven't re-fetched robots.txt yet (wait a few days or submit update in Search Console); ②Incorrect path prefix matching (e.g., Disallow: admin missing /, should be Disallow: /admin/); ③Pages were already indexed before adding Disallow (already indexed pages won't be removed automatically); ④Malicious crawlers don't respect robots.txt; ⑤Allow rule conflict overrides Disallow (Allow takes precedence when path is longer). To completely remove from index, use the noindex tag.
Google Search Console reports that robots.txt is blocking pages?
This usually means important pages are accidentally Disallowed. Check robots.txt for overly broad Disallow rules (such as Disallow: / or Disallow: /*?), confirm that CSS/JS files are not blocked (Google needs to fetch these files to render pages). Use Search Console's robots.txt testing tool to enter specific URLs and test which rule is blocking them.
Accidentally set Disallow: / causing the entire site to be blocked from crawling?
Immediately remove or modify that rule, change robots.txt to Allow: / or delete the Disallow: / line, and upload the updated file to the server. Then submit a robots.txt update request in Google Search Console, and submit a sitemap to accelerate re-crawling. Pages already removed from index may take days to weeks to be re-indexed.
Glossary
- robots.txt
- A plain text file placed in the website root directory, following the Robots Exclusion Protocol, used to tell compliant search engine crawlers which paths are allowed/disallowed for crawling.
- Robots Exclusion Protocol (REP)
- The Robots Exclusion Protocol, proposed in 1994 and standardized as RFC 9309 in 2022, is the de facto standard for communication of crawl rules between websites and crawlers.
- User-agent
- The starting field of a rule group, specifying the crawler name to which subsequent Allow/Disallow rules apply; the * wildcard matches all crawlers.
- Disallow
- Disallow crawling directive, specifying path prefixes you do not want crawlers to access. For example, Disallow: /admin disallows crawling all paths starting with /admin.
- Allow
- Allow crawling directive, specifying paths explicitly allowed for crawler access. Used to open specific subpaths under a Disallowed parent path.
- Sitemap
- Sitemap declaration directive, telling search engines the full URL of the website's XML sitemap, helping crawlers efficiently discover and index all site pages.
- Host
- Preferred domain directive, specifying the site's primary domain (such as example.com vs www.example.com); supported by Yandex, Google uses Search Console for settings.
- Crawler/Bot/Spider
- Automated programs used by search engines to automatically access web pages and collect content, such as Googlebot, Bingbot, Baiduspider, etc.
- Googlebot
- Google search engine's web crawler, responsible for fetching web content for Google indexing and ranking, one of the most common search engine crawlers.
- Crawl Budget
- Crawl budget/quota, the crawl frequency and page number limit allocated by search engines to each website. Properly configuring robots.txt can avoid wasting crawl quota.
- Prefix Matching
- Prefix matching rule, the path matching method used by robots.txt. A match succeeds if the URL path starts with the rule value; longer rules have higher priority.
- noindex
- HTML meta tag or HTTP header directive (X-Robots-Tag: noindex), telling search engines not to include the page in search indexes. Unlike robots.txt Disallow, noindex is a more reliable way to remove from index.
Common Search Engine Crawler User-agent Names
User-agent names used when configuring rules for specific crawlers:
| Crawler Name | Owning Search Engine | Purpose |
|---|---|---|
* | All crawlers | Wildcard, matches all crawlers not separately configured |
Googlebot | Google Search web crawling crawler | |
Bingbot | Bing/Microsoft | Bing Search web crawling crawler |
Baiduspider | Baidu | Baidu Search web crawling crawler |
YandexBot | Yandex | Yandex Search web crawling crawler |
GPTBot | OpenAI | ChatGPT training data crawling crawler |
Twitterbot | X/Twitter | X platform link card preview crawler |
facebookexternalhit | Facebook link preview crawler |
Common robots.txt Rule Examples
Common rule configurations for different site scenarios:
| Rule | Effect |
|---|---|
Disallow: /admin/ | Disallow crawling all content under the /admin/ directory |
Disallow: /*? | Disallow crawling URLs with query parameters (Googlebot supports wildcard *) |
Disallow: /search | Disallow crawling internal site search result pages |
Allow: /public/ | Explicitly allow crawling the /public/ directory |
Disallow: / | ⚠️ Disallow crawling the entire site (fatal error, use with caution!) |
Allow: / | Allow crawling all content on the entire site |
robots.txt Standard Directive Quick Reference
Standard directives and usage defined by RFC 9309:
| Directive | Scope | Format Example | Description |
|---|---|---|---|
User-agent | Group start | User-agent: * | Specifies the crawler name to which rules apply |
Disallow | In group | Disallow: /admin | Path prefix disallowed for crawling |
Allow | In group | Allow: /public | Path prefix allowed for crawling |
Sitemap | Non-group-level | Sitemap: https://example.com/sitemap.xml | Declares sitemap location (absolute URL) |
# | Any position | # This is a comment | Comment line, ignored by crawlers |
Privacy & Security
All operations of this Robots.txt Generator are completed entirely locally in your browser: User-agent, path rules, Sitemap, Host, and other configuration inputs are all assembled in browser memory via JavaScript in real time to generate robots.txt text, and will not be sent over the network to any server. Ready to use once the page loads; does not use Cookie tracking, does not collect any user data. After closing or refreshing the page, all configuration content is automatically cleared and will not be persistently stored in the browser.
Authoritative References
- Authentication Header Generator
- Cache-Control Parser
- Content-Disposition Parser
- CORS Header Generator
- CORS Inspector
- CSP Generator
- cURL to Code Generator
- DNS Propagation Checker
- DNS Lookup
- Forwarded Header Parser
- Hreflang Tag Generator
- HSTS Analyzer
- HTTP Cookie Parser
- HTTP Headers Checker
- HTTP Request Tester
- HTTP Status Codes Lookup
- IP Lookup
- IPv4 Converter
- IPv4 Range Expander
- IPv6 Toolbox
- Link Header Parser
- MX Lookup
- Port Checker
- URL Parameter Builder
- Rate Limit Header Parser
- Redirect Chain Checker
- Robots.txt Generator
- Robots.txt Checker
- Security Headers Checker
- Security.txt Generator
- Set-Cookie Parser
- Site Network Audit
- Sitemap Generator
- Sitemap Inspector
- SSL Certificate Checker
- Subnet Calculator
- URL Parser
- User-Agent Parser
- UTM Link Generator
- WebSocket Tester
- What Is My IP?
- WHOIS Lookup