robots.txt Allow and Disallow Order for Googlebot
If an Allow and a Disallow rule both match a URL in Googlebot's selected group, the more specific match wins. Changing which line comes last does not resolve the conflict.
Worked example
Allow one exact status URL within a blocked path
User-agent: * Disallow: /drafts/ Allow: /drafts/status.html$
Steps
- Choose the rule group that applies to the crawler. For this example use only User-agent: *, so it also applies to Googlebot when no more specific Googlebot group exists.
- Disallow the /drafts/ prefix, then add a more specific Allow for the public status page. The end marker $ restricts the example to the exact URL path without a query string.
- Check the behavior for /drafts/status.html, /drafts/internal.html, and /drafts/status.html?version=2. The first can be crawled; the last two still match the directory block.
What to check
- For equally specific conflicting matches, Google chooses the less restrictive rule. This example relies on the public path being more specific than the directory prefix.
- Check for a Googlebot-specific group in the deployed file: Googlebot does not inherit the wildcard group's rules if a more specific group applies. Protect actual private material with authentication.