Robots.txt
Limit crawler access to duplicate filter paths, but do not use robots.txt as permission for the admin area.
You can say this to an AI Agent
Ready to paste into Cursor, Claude Code, or CodexCreate robots.txt at the site root to limit crawling of duplicate filter paths while keeping normal product pages crawlable. Check the file location, rule matches, and that a normal browser can still access the path. Protect admin pages and customer data with authentication and authorization, not Disallow.