Robots.txt vs Meta Robots vs X-Robots-Tag: The Ecommerce Filter URL Strategy
Ecommerce stores have thousands of filter URLs (color, size, price, brand) that quietly burn 30-60% of Googlebot's crawl budget. Each of the three robots controls solves a different problem — picking the wrong one costs you indexation.
Every ecommerce category page becomes thousands of URLs the moment filters are added. Color, size, price range, brand, material, in-stock status, sort order — each filter combination is a separate URL from Googlebot's perspective. A category with 8 colors × 6 sizes × 5 price ranges × 12 brands × 3 sort orders generates 8,640 URL variants. Multiply by 200 categories and the surface is 1.7 million URLs Googlebot could crawl, almost all of which are duplicate content or near-duplicate.
The three primary tools for controlling what Googlebot sees — robots.txt, meta robots tags, and X-Robots-Tag HTTP headers — each solve a different part of the problem. Picking the wrong one quietly costs you indexation, crawl budget, or rankings on the pages you do want indexed.
This article is a clean breakdown of what each tool actually does, when to use which, and the four configuration patterns that work for ecommerce filter URLs.
What Each Tool Does (Concisely)
robots.txt: tells Googlebot whether to crawl a URL. Does NOT control whether the URL is indexed. If a URL is blocked in robots.txt but other sites link to it, Google can still index it (showing a snippet-less listing with "no information available about this page"). Robots.txt is for crawl budget control, not indexation control.
meta robots tag: <meta name="robots" content="noindex"> in the page <head>. Tells Googlebot the page may be crawled (you have to allow crawling for Google to see this tag) but should not be indexed. The right tool for "I want this page out of search results."
X-Robots-Tag HTTP header: server returns X-Robots-Tag: noindex in the HTTP response headers. Functionally equivalent to the meta robots tag, but works for non-HTML resources (PDFs, images) and can be applied at the server level without touching templates. The right tool for "I want a noindex applied via server rules" and for "this isn't an HTML page."
The fundamental rule that confuses every junior SEO: if you robots.txt-block a URL, Google never crawls it, which means Google never sees the noindex tag inside it. So if your goal is "remove this URL from the index," noindex it and let Google crawl it. Don't robots.txt-block it.
The Four Patterns for Ecommerce Filter URLs
Pattern 1: Index the canonical category, noindex everything else. The simplest, most common, and usually correct default. Your category page /category/shirts/ is fully indexed and canonical. All filter variants — ?color=red, ?size=m, ?price=20-50 — return a page with <meta name="robots" content="noindex,follow">. Google crawls them, sees they're noindexed, eventually deprioritizes crawling them. The follow directive preserves internal link equity.
When to use: most stores under 50,000 SKUs. The few high-value filter variants you do want indexed (e.g., "/category/shirts/blue/") become server-rendered subcategory pages with their own slug, not filter URLs.
Pattern 2: Robots.txt block low-value parameter combinations. The next level up. For stores where Googlebot is wasting crawl budget on parameter combinations even though they're noindexed (Google still crawls noindex pages to verify the tag), you robots.txt-block the parameters that generate combinatorial explosion. Typically: ?sort=, ?view=, ?page= (sort orders never become canonical), and any session parameters.
User-agent: *
Disallow: /*?sort=
Disallow: /*?view=
Disallow: /*sessionid=
Disallow: /*?utm_
When to use: large catalogs (50,000+ SKUs) with measurable crawl budget pressure (Search Console crawl stats show 10×+ more crawl requests than indexed pages).
Pattern 3: Selective canonicalization for valuable filter combinations. Some filter URLs are high-intent and worth indexing — "/category/shirts/blue-cotton/" has search demand for "blue cotton shirts." For these, you either rewrite to a clean URL (preferred) or set the canonical to itself (acceptable) and remove the noindex.
The risk with canonical-only: Google may ignore the canonical if the page looks too similar to the parent. The cleanest pattern is to generate distinct "landing pages" for the filter combinations you want to index, with unique content (introductions, brand stories, FAQs), and use Pattern 1 for everything else.
Pattern 4: X-Robots-Tag at the server for non-HTML or templated rules. For Shopify or WooCommerce stores using CDN-level routing (Cloudflare Workers, Fastly), you can apply X-Robots-Tag: noindex, follow at the edge for any URL matching a pattern. This is faster than template changes (no theme deploy needed) and works for PDF spec sheets, downloadable warranty docs, and other non-HTML resources you don't want indexed.
When to use: enterprise stores with edge compute, or any store with downloadable PDF assets that shouldn't compete with HTML pages in search results.
The Faceted Navigation Specific Problems
Faceted navigation (left-sidebar filter UI on category pages) generates URLs through five mechanisms:
- Query parameters:
/category/shirts/?color=red&size=m. Most common, most controllable. - Path segments:
/category/shirts/red/medium/. Common in WordPress + WooCommerce with custom taxonomies. Harder to control because they look like real category pages. - Anchor fragments:
/category/shirts/#color=red. Modern SPA pattern. Google ignores fragments, so these don't generate crawl URLs — but they also don't get indexed under any condition, so they're invisible to SEO. Use only if you don't want filter URLs to rank. - Sort order parameters:
?sort=price-asc. Always noindex; almost never canonical. - View modes:
?view=gridor?per-page=48. Always noindex.
The path-segment faceted navigation is the trickiest. WooCommerce's WP_Term setup creates real URLs that Googlebot treats as legitimate category pages. You can't robots.txt block them easily (they're in your sitemap; Google trusts the sitemap). The fix is meta robots noindex on the filter-combination pages and only-sitemap-include the canonical category. Plus 301-redirect old filter-combination URLs that previously got indexed.
The Sitemap Question
Don't include noindex'd URLs in your sitemap. The signals contradict each other: sitemap says "this is important, please crawl and index"; meta tag says "don't index this." Google will eventually trust the noindex but it wastes crawl budget in the meantime.
Sitemap should contain only:
- Canonical category and subcategory pages.
- Canonical product detail pages.
- High-value blog and editorial content.
- Static pages (about, contact, FAQ, policies) you want indexed.
Sitemap should NOT contain:
- Filter combination URLs.
- Sort order URLs.
- Paginated category pages 2-N (debatable; modern guidance is to omit them and rely on internal links).
- Out-of-stock product pages with no return ETA.
- Internal search result pages.
The Auditing Workflow
How to audit your filter URL strategy in 60 minutes:
- Run a Screaming Frog crawl with parameter capture. Filter to "Status Code: 200, Content Type: HTML." Note the URL count per parameter pattern.
- Check Search Console "Crawled — currently not indexed." This is where Google reports it's crawling but not indexing. If 10,000+ filter URLs are here, the noindex is working but crawl budget is being wasted — escalate to robots.txt.
- Check "Discovered — currently not indexed." This is where Google sees the URL but hasn't crawled it yet. Sometimes filter combinations sit here forever; Google has given up on them.
- Check "Indexed, though blocked by robots.txt." This is the failure mode of robots.txt-blocking a URL with external links. Solution: unblock, noindex, let Google crawl and de-index naturally.
- Sample 20 filter URLs. View source. Confirm
<meta name="robots" content="noindex">is present. Check curl response headers forX-Robots-Tag. Verify the rendered HTML (not just the source) — some platforms generate the noindex via JavaScript, which Googlebot may not see.
The Common Configuration Mistakes
- Robots.txt blocking pages you want de-indexed. Google can't see the noindex if it can't crawl. The URLs remain in the index forever with no snippet. Fix: unblock, noindex, let Google crawl, wait 4-8 weeks for de-indexation.
- Noindex on the canonical category page. Easy mistake when the noindex rule is "/category/*?*" with a broken regex that matches the parent. Audit Search Console weekly.
- nofollow on filter URLs. Cuts internal PageRank flow. Use noindex,follow not noindex,nofollow.
- Different canonical from the URL itself on the canonical page. Means you're telling Google the canonical is somewhere else; Google ignores your URL. Self-referential canonicals are correct for canonical pages.
- X-Robots-Tag on HTML pages where the template also has a meta robots tag. Conflicting signals confuse Googlebot. Pick one.
What StoreVitals Detects
Our crawler flags:
- Pages with conflicting robots signals (e.g., noindex meta + index in HTTP header).
- Sitemap URLs that return noindex (sitemap pollution).
- Canonical URLs pointing to themselves with noindex (the contradictory pattern).
- Robots.txt rules that block sitemap URLs.
- X-Robots-Tag presence in HTTP headers.
- Faceted navigation URL count via crawl depth analysis (when our 50-page crawl finds 20+ parameter-only variants of a single category, we flag it).
Run a free scan to audit your robots configuration. For deeper crawl budget analysis on large catalogs (100,000+ URLs), pair StoreVitals with a Screaming Frog or JetOctopus crawl — different tools for different scales.