robots.txt in 2026: Rules, Edge Cases, and Real-World Examples

Reference

robots.txt looks simple. That is exactly why it causes expensive mistakes.

A single character can change crawl access across an entire site. A rule that seems correct in theory can behave differently in practice because crawlers follow a matching algorithm, not human intent. And even when a rule works exactly as written, it may still fail to achieve the outcome you actually want, because robots.txt controls crawling, not indexing.

In 2026, robots.txt is no longer just a loose convention. The Robots Exclusion Protocol is now formalised as RFC 9309, which clarifies how files are parsed and reinforces a key point: robots.txt is advisory, not access control.

This guide focuses on how robots.txt behaves in the real world:

• what it can and cannot do
• how rules are matched and resolved
• where teams make costly assumptions
• and practical examples you can adapt safely

If you want a quick way to draft and sanity-check a file, this free generator is useful for validation and edge-case testing:
https://twosquares.co.uk/tools/robots-txt

---

What robots.txt actually does

robots.txt provides crawl directives to automated agents that choose to respect the protocol. It must be publicly accessible at:

https://example.com/robots.txt

When fetched, it tells compliant crawlers which URL paths they should not fetch. The key word is should. It is guidance, not enforcement.

What robots.txt does well

• Prevents wasteful crawling of duplicate or low-value URL spaces
• Helps manage crawl load on large or complex sites
• Communicates crawl priorities indirectly
• Reduces crawling of internal search, filters, and parameters

What robots.txt does not do

robots.txt does not:

• secure private content
• block access for humans
• guarantee de-indexing
• override public availability

If a URL is publicly accessible, it can still be discovered through links, referrers, sitemaps, or third-party sources. Blocking crawling does not make a URL invisible.

This distinction matters more now than ever.

---

Crawl control vs index control (the critical difference)

A common mistake is using robots.txt to try to remove pages from search results.

robots.txt only affects crawling. Indexing is a separate decision.

If a page is blocked by robots.txt:

• Google may still index the URL
• the page can appear as a “URL-only” result
• Google cannot see on-page directives like noindex

If your goal is do not index, blocking crawling often works against you.

A simple rule that avoids many problems:

• Use robots.txt to control crawling
• Use noindex (meta or HTTP header) to control indexing

They solve different problems and are not interchangeable.

If you’re deciding between crawl control, index exclusion, and consolidation, use the decision framework in noindex vs canonical vs robots.txt.

---

robots.txt structure and parsing

A robots.txt file is made up of one or more groups.

Each group contains:

• one or more User-agent lines
• followed by rules that apply to that agent

Example:

User-agent: *
Disallow: /private/

Crawlers select the most specific matching group for their user-agent string. If multiple groups match, the most specific one is used.

How path matching really works Path matching is prefix-based.

Disallow: /blog/ Blocks:

/blog/

/blog/post-1

/blog/category/example

Does not block:

/blogs/

/my-blog/

Modern crawlers also support:

• as a wildcard

$ as an end-of-URL anchor

These are not part of the original 1994 convention, but are widely supported in practice.

Allow vs Disallow: conflict resolution One of the most misunderstood behaviours is how conflicting rules are resolved.

When multiple rules match a URL:

the most specific match wins

specificity is generally determined by path length

Example:

User-agent: * Disallow: /blog/ Allow: /blog/launch-post.html Result:

/blog/ is blocked

/blog/launch-post.html is crawlable

This works because the Allow rule is more specific than the Disallow rule.

Do not assume “Allow always overrides Disallow”. Specificity matters.

Wildcards: powerful and dangerous Blocking tracking parameters User-agent: * Disallow: /?utm_ Disallow: /&utm_ Disallow: /?gclid= Disallow: /&gclid= Disallow: /?fbclid= Disallow: /&fbclid= This prevents crawling of tracking variants that generate duplicate URLs.

Important nuance:

This reduces crawl waste

It does not consolidate ranking signals

Canonicals and consistent internal linking still matter

Blocking file types User-agent: * Disallow: /*.pdf$ Blocks:

/docs/guide.pdf

Does not block:

/docs/guide.pdf?download=true

If PDFs are part of your search strategy, blocking them may be counter-productive.

Sitemap directives You can include sitemap locations in robots.txt:

Sitemap: https://example.com/sitemap.xml
Sitemap: https://example.com/sitemap-products.xml

This is not access control. It is a discovery hint.

Most major crawlers support it, and it helps align crawling with your preferred URL sets.

crawl-delay: use with caution Crawl-delay is not part of the core robots standard and support varies.

Some crawlers interpret it. Others ignore it entirely.

In practice:

It is rarely respected by Google

It can significantly slow discovery

It often masks underlying problems

If crawl rate is causing issues, better solutions usually include:

fixing infinite URL spaces

reducing parameter duplication

improving caching and server performance

crawl-delay is a blunt instrument.

Real-world robots.txt examples Sensible default for many sites User-agent: * Disallow: /wp-admin/ Disallow: /cart/ Disallow: /checkout/ Disallow: /my-account/ Disallow: /search/ Disallow: /?s= Disallow: /*?s= Allow: /wp-admin/admin-ajax.php

Sitemap: https://example.com/sitemap.xml

This:

blocks low-value or sensitive paths

preserves required front-end functionality

exposes sitemap locations clearly

Blocking staging environments User-agent: * Disallow: / This tells crawlers not to crawl anything.

It does not secure the site.

For staging, proper protection (authentication, IP restrictions) is still required.

Faceted navigation control (ecommerce) User-agent: * Disallow: /?color= Disallow: /&color= Disallow: /?size= Disallow: /&size= Disallow: /?price= Disallow: /&price= Disallow: /?sort= Disallow: /&sort= Important trade-off:

This reduces infinite crawl paths

It can also block useful category combinations

A common pattern is:

allow a curated set of indexable facet URLs

block the rest

robots.txt vs canonicals vs noindex robots.txt is often used where it should not be.

Quick decision guide:

Want to stop crawling? → robots.txt

Want to stop indexing? → noindex

Want to consolidate duplicates? → canonical

Want security? → authentication

Using the wrong tool usually creates silent failure rather than visible errors.

Common mistakes that still happen Leaving Disallow: / live after launch

Blocking CSS or JS required for rendering

Over-broad wildcard rules

Assuming all crawlers behave the same way

Using robots.txt as a privacy mechanism

Most of these mistakes do not throw errors. They quietly degrade performance.

Testing before deployment robots.txt should always be treated as production-critical.

Before publishing changes:

test rules against real URLs

check Allow vs Disallow conflicts

verify sitemap URLs resolve correctly

ensure /robots.txt returns a 200 status

A lightweight way to do this is with a generator and validator that shows matches explicitly, such as: https://twosquares.co.uk/tools/robots-txt

Summary robots.txt is deceptively simple syntax sitting on top of complex crawler behaviour.

Used well, it:

reduces crawl waste

protects intent boundaries

supports scalable SEO

Used carelessly, it:

blocks valuable content

hides problems instead of solving them

causes silent, long-lasting damage

In practice, the best robots.txt files are:

conservative

readable

reversible

documented

Clarity beats cleverness every time.

---

Related reading

Glossary terms

• Internal Linking

• Access Control

• Crawl Budget

• noindex vs canonical vs robots.txt: how to choose

• URL parameters at scale

• XML sitemaps in 2026

• Crawl budget in 2026

• Free SEO audit