ai-crawler-checklisted

Diagnose whether a website is accidentally blocking AI search crawlers (GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended). Use this whenever a user asks if AI can read/find/cite their site, whether ChatGPT or Claude or Perplexity can see their pages, why their site isn't showing up in AI search or AI answers, or asks to check/audit their robots.txt for AI or LLM crawler access. Trigger it for phrases like "can ChatGPT read my site", "is my site blocking AI", "check my robots.txt for AI crawlers", "why don't I show up in AI search", or when someone shares a campaign/business URL and worries about AI visibility. Diagnostic only — it inspects robots.txt and headers and reports findings plus a suggested fix; it never modifies the user's site.
meturley/trail-marker-geo · ★ 0 · AI & Automation · score 70

Install: claude install-skill meturley/trail-marker-geo

# AI Crawler Check Diagnose whether a website is accidentally blocking the crawlers that AI search products use to read and cite pages. Report findings per bot and give the exact `robots.txt` edit to fix any problem. **This skill is diagnostic only. It never changes the user's site.** It reads `robots.txt` and HTTP/meta directives, then presents findings and a suggested fix that the user applies themselves. ## When to use Use whenever someone wants to know if AI systems can access, read, index, or cite their website — especially campaign, small-business, or personal sites worried about AI-search visibility. Also use for "audit my robots.txt for AI crawlers" type requests. ## What you need from the user Just the website URL (e.g. `example.com` or `https://example.com`). If they haven't given one, ask for it. One domain per run. ## How to run it The bundled script does the fetching and evaluation. Run it against the URL: ```bash python3 scripts/check_crawlers.py <URL> ``` Add `--json` if you want structured output to reformat yourself. The script: 1. Fetches `<site>/robots.txt` (a missing/404 robots.txt means everything is allowed). 2. Fetches the homepage to inspect the `X-Robots-Tag` response header and any `<meta name="robots">` tags. 3. Evaluates every known AI crawler user-agent against the robots.txt rules using the standard longest-match algorithm (so `Allow:` can override a broader `Disallow:`, and an explicit per-bot group overrides the `*` group). 4