# Toeshee robots.txt — https://toeshee.com # Policy: OPEN to search engines and AI engines alike, with the AI crawlers # marketing cares about explicitly allowed. Query-string URLs are excluded from # crawling (on-page search results and campaign parameters — thin/duplicate # content). Abusive bots are handled by server-side rate limiting + reCAPTCHA. # --- AI engine crawlers: explicit allow --- User-agent: GPTBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: Google-Extended Allow: / User-agent: ClaudeBot Allow: / User-agent: anthropic-ai Allow: / User-agent: Claude-Web Allow: / User-agent: bingbot Allow: / User-agent: Meta-ExternalAgent Allow: / User-agent: CCBot Allow: / # --- Everyone else --- User-agent: * Allow: / Disallow: /portal.html Disallow: /previews/ Disallow: /? Disallow: /*?* Sitemap: https://toeshee.com/sitemap.xml