Saturday, November 21, 2026

Top 5 This Week

Related Posts

Google: robots.txt prioritizes precise rules | Brief AI


In robots.txt, Google applies the rule corresponding to the most characters in a URL and favors authorization in case of a tie. This file governs the exploration of sites by search engine robots and AI agents. Poorly targeted restriction can exclude content from search results or major language models.

Google needs to read the X-Robots-Tag: noindex HTTP header

Preventing a robot from viewing a PDF does not serve the same purpose as requesting its exclusion from the results. For this last operation, Google must be able to read the HTTP header X-Robots-Tag: noindex: the document must therefore remain accessible for exploration. This distinction concerns in particular commercial guides, brochures and other files offered for download.

The Disallow: /*.pdf$ directive targets addresses ending in .pdf, regardless of the directory. However, it allows /guide.pdf?download=1 as well as the extensions written .PDF. Each variation requires separate verification.

In WordPress, Disallow: /wp-content/uploads/*.pdf$ allows you to target documents in the upload folder. An exception such as Allow: /wp-content/uploads/2024/09/allowed-document.pdf$ nevertheless allows this specific document. Its more detailed path trumps the general ban.

The rules applicable to each robot

Installed at the root of the site, robots.txt is a text file intended to supervise the visits of robots, now both those of engines and AI agents. User-agent designates the recipient of the instructions; the group marked with the asterisk serves as a default when no particular group matches. Disallow asks file-friendly bots to avoid certain paths, while Allow provides permissions, including under a prohibited folder. The Sitemap directive provides the address of the XML plan to help engines find it.

A robot-specific group replaces the general instructions instead of complementing them. Googlebot can thus visit /private/ if no ban appears in its group, even if this path is blocked elsewhere: the restriction must be placed in the group that corresponds to it. For AI training robots, the choice of prohibitions must take into account documented uses; accesses triggered at the request of users must be examined separately.

In the same group, Allow: /downloads/free/ preserves access to this subfolder despite Disallow: /downloads/, which continues to target the rest. If the Allow and Disallow directives refer exactly to /downloads/, Google allows the visit. The precision of the path therefore decides between the rules before their more or less restrictive nature.

The syntax does not offer all the possibilities of regular expressions: only two wildcard characters are available. The asterisk accepts zero or more characters; the dollar marks the end of the address. For paths, filter= differs from Filter=, unlike directive names and agent IDs, which are case insensitive.

The evening letterFree · 7 p.m.

Do you want the best AI tools before the rest?

We test and decipher the new AI tools every evening, in 5 minutes. Free.

Included upon registration: our selection of the best AI guides & comparisons.

Choose your pace

Free · No spam · Unsubscribe in 1 click

Limit unnecessary visits and protect accounts

On a merchant catalog bringing together hundreds of thousands of pages, filters can endlessly multiply the variants, without their own content. In the absence of restrictions, Google can try combinations of parameters even on non-existent addresses, with peaks in requests and unnecessary consumption of server and robot resources. Google’s Gary Illyes has repeatedly warned of this risk for action addresses. In 2024, Google updated its indications, particularly regarding additions to the basket or wish lists.

Limiting visits to pages without desired visibility, such as a privacy policy, can encourage exploration of useful products and services. Google specifies, however, that the resources saved are not automatically transferred elsewhere, except when the site already reaches its crawling capacity limit. Avoiding unnecessary visits also reduces the calculation required for motors, with a sustainability benefit.

The selection depends on the usefulness of the pages and the indexing objectives: internal searches, sorting without an SEO objective and commercial actions do not all justify a visit. Non-result account spaces may also be restricted, but authentication is still necessary to protect private information. JavaScript scripts with no role in content or rendering, including certain tracking tools, may also be affected.

For example, a form can pass its data to an API endpoint with the action=”/form/submissions/” attribute. If a robot discovers this address, Disallow: /form/ asks not to explore it. The restriction is on pages under this directory without the need for an asterisk.

Choose the filters to leave accessible

Not all merchant facets are intended to be blocked. Zalando thus has pages by color targeting in particular “gray t-shirt”, a use of filters aimed at general product searches. Google distinguishes these useful destinations from undesirable combinations: the former must remain searchable. Conversely, filters limited to sorting products can result in dozens of duplicate pages.

As an address can bring together several parameters, each criterion to be restricted must be recognized in different locations. For sortby, the patterns are /?sortby= and /?*&sortby= ; the same principle applies to color and price. Only those criteria that are actually undesirable should be added to the bans, with tests covering both the addresses to keep and those to exclude.

UTM parameters have another function: tracking. John Mueller discussed tracking parameters in external links on Reddit, a situation separate from the filters and searches produced by the site itself. A string of parameters is therefore not enough to justify blocking: its use on the site counts. Internal links must point to the addresses you want to visit.

Restrict internal searches on WordPress

Pages from the internal search engine may be restricted when they are not intended to appear in the results of external engines. In WordPress, this search typically relies on s, as in https://www.example.com/?s=google. These results constitute a frequent source of unnecessary visits.

To target them, Disallow: /?s= recognizes the initial parameter, while Disallow: /?*&s= recognizes it after another. These paths start with / and avoid aiming for a different name where only the last letter would be s. They do not recognize /?S=: the verification must concern the spelling and the order of the parameters actually used.

Shredder Smith
Shredder Smith
Shredder Smith is the lead curator and digital persona behind topaitools4you.com, an AI directory dedicated to "shredding" through industry hype to identify high-utility software for everyday users. Smith positions himself as a blunt, no-nonsense reviewer who vets thousands of emerging applications to filter out overpriced "wrappers" in favor of tools that offer genuine ROI and practical productivity. The site serves as a watchdog for the AI gold rush, providing categorized rankings and transparent reviews designed to help small businesses and creators navigate the crowded tech landscape without wasting money on low-value tools.

LEAVE A REPLY

Please enter your comment!
Please enter your name here

Popular Articles