robots.txt Allow and Disallow Order for Googlebot

If an Allow and a Disallow rule both match a URL in Googlebot's selected group, the more specific match wins. Changing which line comes last does not resolve the conflict.

Worked example

Allow one exact status URL within a blocked path

User-agent: *
Disallow: /drafts/
Allow: /drafts/status.html$

Steps

  1. Choose the rule group that applies to the crawler. For this example use only User-agent: *, so it also applies to Googlebot when no more specific Googlebot group exists.
  2. Disallow the /drafts/ prefix, then add a more specific Allow for the public status page. The end marker $ restricts the example to the exact URL path without a query string.
  3. Check the behavior for /drafts/status.html, /drafts/internal.html, and /drafts/status.html?version=2. The first can be crawled; the last two still match the directory block.

What to check

  • For equally specific conflicting matches, Google chooses the less restrictive rule. This example relies on the public path being more specific than the directory prefix.
  • Check for a Googlebot-specific group in the deployed file: Googlebot does not inherit the wildcard group's rules if a more specific group applies. Protect actual private material with authentication.

References