← ClaudeAtlas

scrape-batchlisted

Extract many known URLs in one polite, rate-limited pass. Use when the user hands over a list of links, a set of search hits to read in full, or asks to "scrape these pages" / "pull the content from all of them". Drives extract(action="batch"), which fans out with per-domain rate limiting and returns partial results plus a per-URL error list.
n24q02m/wet-mcp · ★ 16 · AI & Automation · score 80
Install: claude install-skill n24q02m/wet-mcp
# scrape-batch Fan out `extract(action="batch")` over a URL list wet already knows, then report per-URL outcomes honestly. The batch path applies per-domain politeness (2 concurrent and 1 request/second per domain, 6 fetches in flight overall) and returns whatever succeeded even when some URLs fail. Use this skill when: - The user supplies a list of URLs to read in full. - A previous `search` returned hits and the user wants the bodies, not the snippets. - A crawl or `extract(action="map")` produced a URL set to pull down. Do NOT use this skill when: - There is one URL, or a handful from one domain -- call `extract(action="extract", urls=[...])`, which is cached and cheaper. - The URLs are not known yet and the goal is an answer, not the pages -- use the `research-topic` skill (`extract(action="agent")`). - The target is a whole site rather than a list -- use `extract(action="crawl")` or `extract(action="map")`. - The user wants images or video from the pages -- use `media(action="list")` then `media(action="download")`. ## Steps 1. **Collect and de-duplicate the URL list.** Drop duplicates and fragment-only variants (`#section`) -- each one costs a full fetch. Report the final count to the user before spending it. 2. **Split into chunks of at most 50.** The cap is hard: 51 URLs returns `{"error": "Error: Maximum 50 URLs per batch (got 51)"}` and nothing is fetched -- the call is refused, not truncated. 3. **Split further when one domain dominates