Robots.txt

You might say

Limit crawler access to duplicate filter paths, but do not use robots.txt as permission for the admin area.

A root-level file that tells automated crawlers which paths they may crawlShopping carts and duplicate filter URLs can waste crawler requests, so robots.txt can describe crawling rules at the site root. It is guidance for crawlers that follow the rules, not a block on normal visitors, authentication, or protection for sensitive data.
Robots Exclusion Protocol

You can say this to an AI Agent

Ready to paste into Cursor, Claude Code, or Codex
Structured AI Agent prompt

Create robots.txt at the site root to limit crawling of duplicate filter paths while keeping normal product pages crawlable. Check the file location, rule matches, and that a normal browser can still access the path. Protect admin pages and customer data with authentication and authorization, not Disallow.

Want your Agent to translate plain-language UI ideas automatically?Install free VibeHub Skill →
Further reading