crawlability-auditlisted

# Crawlability Audit ## Install Save this file as `~/.claude/skills/crawlability-audit/SKILL.md`, or `.claude/skills/crawlability-audit/SKILL.md` to scope it to one repo. Claude Code auto-discovers it. Invoke with `/crawlability-audit` or by asking "why isn't this page getting indexed?". ## Why this exists SEO advice is mostly about content. The failures that actually cost the most are mechanical, silent, and take minutes to fix once seen: a prefix rule that blocks more than anyone intended, a sitemap advertising URLs that robots forbids, an error page returning 200 so crawlers file it as thin content instead of retrying. None of these announce themselves. Traffic is simply flat, and everyone blames the copy. Rank by what it costs. A rule blocking your signup or seller page outranks a missing meta description on a blog post, always. ## Step 1 — Read robots.txt as a machine would ``` curl -s https://<host>/robots.txt ``` **Every `Disallow` is a PREFIX match, not a path match.** This is the single most expensive misunderstanding in the file: - `Disallow: /sell` blocks `/sell`, and also `/sellers`, `/selling`, `/seller/anyone`, `/sell-your-thing`. Whole sections vanish. - `Disallow: /sell/` blocks only children, leaving `/sell` itself crawlable. For each `Disallow`, enumerate the real routes it matches. Read the router or route directory; do not guess: ``` rg --files -g '*/page.tsx' -g '*/route.ts' -g 'pages/**' | sort ``` Then ask, per matched route: **is this ge