docs-crawler

# docs-crawler Crawl a documentation website into a single Markdown corpus that an LLM can consume as context. ## When to use The user wants to capture a multi-page documentation, design-system, or reference site as Markdown — to feed an LLM, archive it, or research a design system. This skill is also invoked by the `design-md` skill's research phase. ## Inputs - **Site URL** (required) — the documentation site, e.g. `https://socarframe.socar.kr/`. Any page on the site works; page discovery starts from the site's `sitemap.xml`. - **Output directory** (optional) — where to write the corpus. Defaults to `./{host}-docs/` in the current working directory. If the user has not provided a URL, ask for one — do not guess. ## Running the crawl From the repository root: ``` pnpm crawl:docs <site-url> [--out <dir>] [--external-images] ``` Equivalent direct form: ``` pnpm exec tsx .claude/skills/docs-crawler/scripts/crawl.ts <site-url> [--out <dir>] ``` The engine discovers pages via `sitemap.xml` (falling back to same-origin link-following), fetches each page, extracts the main content, and converts it to Markdown. It prints per-page progress and a final summary line. By default it also **localizes images**: every external image and inline base64 `data:` image is downloaded into `crawl/images/` and the Markdown is rewritten to relative paths, so the corpus is self-contained for renderers that can't fetch external URLs (Claude Design, offline previews). Pass `--externa...