A store wants Google to discover its products, avoid endless filter combinations, and respect a decision about AI training. The team adds several bot names to robots.txt and assumes the job is finished.
Those goals involve different controls. A crawl restriction does not remove an indexed page, a training preference does not govern every AI product, and a user-requested visit may follow different rules from an automatic crawler.
Robots.txt is a public text file that communicates crawling rules to supporting automated clients. Use it to manage intended crawler access, then verify the published response and the behavior relevant to your goal.
This guide explains the rules through practical examples and a deployment checklist. It is a decision framework, not a universal file to paste over your existing configuration.
Documentation checked November 19, 2026.
What Is Robots.txt, in Plain English?
Think of robots.txt as instructions at a building’s reception desk: “Delivery teams should not use this corridor.”
A cooperating visitor can follow that instruction. The sign does not lock a door or remove the building from a directory.
Google’s robots.txt introduction explains that the file manages crawler traffic. It should not be treated as a reliable way to keep a webpage out of Google Search.
A blocked URL can still appear based on information Google finds elsewhere, such as links.
For broader context, technical SEO also covers indexing instructions, canonical URLs, rendering, and availability. Robots.txt handles one part of that system.
Where Does the File Belong?
Google’s file creation guide places it at the site’s root:
https://example.com/robots.txt
Use plain text and UTF-8 encoding. Your CMS or hosting provider may generate the file through settings rather than a physical file you edit directly.
Its scope matters. The file served on one protocol, host, and port does not automatically govern every subdomain or alternative address.
Inventory the public hosts your business actually uses. A main website, a shop subdomain, and a separate documentation host deserve individual checks.
Choose the Outcome Before Choosing the Rule
Write a plain-language objective first.
Reduce unwanted crawling: Identify the URL patterns generating unnecessary requests. Robots.txt may be appropriate for supporting crawlers.
Keep an accessible page out of Google: Use a supported indexing treatment rather than assuming a crawl block removes it.
Protect private information: Require authentication and proper application permissions.
Restrict a documented AI use: Identify the provider, product, and supported preference.
Support a user-requested agent task: Review the actual infrastructure and application access path.
This short decision step prevents one file from being asked to do five different jobs.
Robots.txt and Noindex Solve Different Problems
Google’s noindex documentation explains that a crawler must be able to access a page to read its noindex tag or response header.
Blocking that page in robots.txt can prevent Google from seeing the instruction.
For example, if an internal report should remain publicly accessible but not appear in Google, review the appropriate noindex implementation. If it contains confidential data, public access itself needs addressing.
Google does not support a noindex directive placed inside robots.txt.
When reviewing index bloat, distinguish unwanted indexed URLs from unwanted crawling. They may overlap, but the treatment must match the problem.
Do not publish sensitive URLs merely to let a crawler read an exclusion instruction.
Understand User-Agent, Disallow, Allow, and Sitemap
The Robots Exclusion Protocol defines the standard foundation. Google documents its own interpretation and extensions.
A basic example for a hypothetical site is:
User-agent: *
Disallow: /catalogue-search/
Allow: /catalogue-search/help/
Sitemap: https://example.com/sitemap-index.xml
Here:
- User-agent identifies the requestor group.
- Disallow restricts matching paths.
- Allow permits matching paths, including a more specific exception.
- Sitemap identifies the actual sitemap location for supporting search engines.
The asterisk supplies fallback rules where no specific group applies. It is not proof that every automated service will obey the file.
In this example, a supporting crawler should avoid catalogue-search URLs while the more specific help directory remains available.
Use your real sitemap address. Publishing a plausible filename does not create the sitemap.
Specific Bot Groups Can Change the Result
Google’s robots.txt specification explains that it selects the most specific matching user-agent group. A specific group does not inherit the wildcard group’s restrictions.
Consider this hypothetical configuration:
User-agent: *
Disallow: /preview-room/
User-agent: Googlebot
Disallow: /campaign-drafts/
For Googlebot, the specific group governs the result. The wildcard restriction on /preview-room/ is not automatically added.
If both restrictions are intended for Googlebot, include both in its applicable group.
Do not rely on moving a group to the top of the file. For Google, group order does not decide precedence.
Match Specificity Matters More Than Line Order
For applicable path rules, the most specific match wins. Google chooses the less restrictive rule when equally specific allow and disallow rules conflict.
Path matching is case-sensitive. A rule for one spelling is not a substitute for checking the URL variants your application actually serves.
These details are why a readable file can still produce an unexpected result.
Before publishing a bot-specific group, write down which existing restrictions it must preserve.
Avoid Treating Robots.txt as Full Regex
Robots rules do not provide the full regular-expression features you may use in Search Console or programming tools.
The standard supports wildcard and end-of-match behavior. For example:
Disallow: /manuals/*.pdf$
This matches relevant URLs ending in .pdf under that path. A URL with a query string after the extension needs separate consideration.
Test real addresses rather than assuming a rule covers every visual variation of a URL.
For parameter-based restrictions, check the parameter’s position and actual spelling. A broad pattern aimed at “search” can accidentally cover a different parameter or leave an alternative form untouched.
Keep examples in your review record. The reviewer should be able to explain why each wanted URL remains reachable.
Decide Which Filter URLs Deserve Search Access
An ecommerce site might produce thousands of sort and filter combinations. Some may be useful landing pages; others may repeat the same inventory in unhelpful arrangements.
Start with URL purpose:
- Does this destination serve a distinct customer need?
- Is it part of the intended search strategy?
- Does it offer a meaningful result?
- How is the application linking to it?
- What do request records show about its use?
Do not block every query string simply because it is a query string.
For a hypothetical clothing store, a deliberately maintained category for waterproof jackets has a different purpose from arbitrary sorting variations. Document that difference before proposing a pattern.
This is a review method, not a claim that one crawl restriction will automatically improve rankings.
Separate AI Search Crawlers From Training Crawlers
OpenAI’s crawler documentation distinguishes several roles:
- OAI-SearchBot: used for ChatGPT search.
- GPTBot: associated with material that may be used for foundation-model training.
- ChatGPT-User: used for certain user-requested actions rather than automatic web crawling.
The documented search and training settings are independent. A publisher can allow search crawling while disallowing GPTBot.
A minimal illustration of that preference is:
User-agent: GPTBot
Disallow: /
This is a policy illustration, not a replacement for your file. Preserve your intended restrictions for other requestors and verify the provider’s current instructions.
OpenAI notes that robots.txt may not apply to user-initiated ChatGPT-User actions.
Access permission also does not guarantee that a provider will retrieve, select, cite, or send traffic to your page.
Google-Extended Is a Different Kind of Token
Google’s common crawler documentation describes Google-Extended as a product-control token without its own separate HTTP user-agent string.
Its documented scope includes specified Gemini training and grounding uses. Google states that it does not affect inclusion or rankings in Google Search.
Therefore, do not search your logs for a standalone Google-Extended visit and treat its absence as proof of compliance or noncompliance.
The broader AI training control discussion also requires attention to product scope and implementation dates. A future shared preference should not be presented as already enforced everywhere.
User-Requested Agents Need a Separate Access Review
Google’s user-triggered fetcher guidance says these fetchers generally ignore robots.txt because a user initiated the request.
This means a crawler policy and a verified agent-access policy are not interchangeable.
If a business wants an agent to use its public search or availability controls, test that interaction through the CDN, firewall, and application. If access should be restricted, implement appropriate access controls.
Do not disable authentication because a request identifies itself as an agent.
Keep the outcome precise: a permitted request is not a completed booking, purchase, or AI citation.
Verify the Published Response, Not Just the Editor
After a change, request the exact public robots.txt address for each affected host.
Record the status code and returned body. A CMS preview cannot prove that a CDN, redirect, or server rule is delivering the intended file.
Check for:
- An unexpected login page.
- An HTML error returned in place of plain text.
- Old rules served from cache.
- A redirect to a different host.
- A server error affecting the file.
Google documents special handling for failed robots.txt retrieval. Server errors can pause crawling, with subsequent behavior depending on cached rules and availability. Most 4xx responses other than 429 are treated as an absent file.
Use the HTTP status-code troubleshooting workflow to distinguish a valid response from an accidental failure.
Build a URL Acceptance Checklist Before Deployment
For every proposed rule, identify examples expected to be allowed and examples expected to be restricted.
For the earlier catalogue-search example, a review could include:
- A normal search-results URL.
- A help page inside the allowed exception.
- A public product page outside the restricted directory.
- A case or parameter variation actually produced by the application.
- A required page resource, where relevant.
Test the named bot and the fallback behavior. Do not stop after checking one URL with a wildcard user agent.
Keep the expected outcome beside the observed result. If they differ, resolve the rule or the expectation before publishing.
Include a Business-Critical Counterexample
A useful test tries to disprove the proposed change.
If the goal is to restrict internal search, include a public product URL that must remain crawlable. If the goal is to preserve an exception, include a neighboring path that should stay restricted.
This turns review into a concrete comparison rather than “the file looks correct.”
For a migration, check the old and new hosts separately. For a CMS change, confirm which component now generates the file.
Observe Requests Without Confusing Them With Citations
After deployment, review relevant request evidence and the reports appropriate to your objective.
Verify bot identity using the provider’s supported method before making infrastructure exceptions. A copied user-agent string is not sufficient authentication.
Separate the observations:
- The intended file was served.
- A verified requestor fetched it.
- The relevant page was requested.
- The response was available.
- A later search or AI outcome was observed.
Our discussion of AI crawlers accessing sitemaps and RSS explains why discovery-file requests should not be reported as proof of citations.
If the goal was deindexing, check indexing outcomes. If it was reducing unnecessary requests, inspect the relevant patterns and time window. Choose the measurement that answers the original question.
A Practical Handover for Your Team
Keep a short record with:
- The business objective.
- Affected hosts and requestors.
- The proposed rules and owner.
- Allowed and restricted URL examples.
- The published response checks.
- Monitoring and rollback responsibilities.
Document unresolved questions explicitly.
For example, “training restriction published; live answer retrieval policy reviewed separately” is more useful than “AI bots blocked.”
Revisit the policy when the website architecture changes or a provider changes its documented behavior.
Frequently Asked Questions
Does every website need a complicated robots.txt file?
No. Add restrictions for a defined purpose. More lines do not establish better SEO.
Can robots.txt remove a page from Google?
It is not a reliable removal method. Use the appropriate indexing or access treatment for the intended outcome.
Does Googlebot inherit User-agent: * restrictions?
Not when a more specific Googlebot group applies. Preserve any necessary restrictions in the applicable group.
Can I restrict AI training while allowing AI search?
Some providers document independent settings. OpenAI distinguishes GPTBot from OAI-SearchBot; review each provider’s current controls.
Is robots.txt enough to secure private pages?
No. Use authentication and application permissions. The file is public and does not enforce confidentiality.
Does allowing a crawler guarantee an AI citation?
No. Access, retrieval, selection, citation, and website visits are separate outcomes.
Make Every Rule Explainable
A useful robots.txt policy has a clear purpose, preserves intended access, and passes checks against real URLs.
Begin with the outcome, use the documented control, and verify what changed. That is more reliable than collecting bot names and hoping one file governs every use.
If you want technical fixes, useful content, and authority-building work managed as a consistent monthly process, connect with Paul Gautheron on LinkedIn. Scale Xpert brings those activities into one visibility workflow, with progress reviewed against meaningful website outcomes.




