Robots.txt

You might say

Limit crawler access to duplicate filter paths, but do not use robots.txt as permission for the admin area.

A root-level file that tells automated crawlers which paths they may crawlShopping carts and duplicate filter URLs can waste crawler requests, so robots.txt can describe crawling rules at the site root. It is guidance for crawlers that follow the rules, not a block on normal visitors, authentication, or protection for sensitive data.

Know first

URL Authorization SEO

Robots Exclusion Protocol

How one rule affects crawlers without becoming a site lock

robots.txt

1User-agent: *

2Disallow: /cart

3Disallow: /admin

CrawlerAllowed to crawl/products

User browserCan still request it200 OK

Disallow is not access control; private pages still need authentication and authorization.

How robots.txt is used

The file location sets its scope: robots.txt normally lives at the site root, such as https://example.com/robots.txt. Rules apply to the matching protocol, host, and port.

A crawler reads the rules before requesting: A compliant crawler uses User-agent, Allow, and Disallow to decide whether to request a path. A normal browser does not lose access because of this file.

Quick checkChoose the best answer

An admin page contains customer data, and the team wants to protect it with Disallow in robots.txt. What is the right judgment?

Protect the admin page with authentication and authorization, then use robots.txt only to reduce crawling if usefulDisallow is enough because a path hidden from crawlers cannot be accessed by anyoneRemove the admin page from navigation instead of adding access control

You can say this to an AI Agent

Create robots.txt at the site root to limit crawling of duplicate filter paths while keeping normal product pages crawlable. Check the file location, rule matches, and that a normal browser can still access the path. Protect admin pages and customer data with authentication and authorization, not Disallow.

Learn next

1Authorization→2XML Sitemap→3SEO→