← ClaudeAtlas

cdp-graphql-scraperlisted

Scrape data from sites you're authorized to access by attaching Playwright to your OWN already-logged-in Chrome over the DevTools protocol (CDP) and harvesting the JSON/GraphQL responses the page already fetches — instead of re-authenticating, driving login forms, or hammering endpoints directly. Use this whenever a task involves collecting structured data from a logged-in web app, building a resilient web scraper/collector, intercepting XHR/GraphQL/API responses, or when an existing scraper is getting rate-limited, soft-blocked, or returning empty results. Also reach for it for the operational side — running, resuming, or debugging a long crawl, or making one survive blocks and restarts. Covers the anti-block hygiene (human-paced delays, batch cooldowns, block-signal detection, resumable state) that naive scrapers miss.
brightstone111/web-scraping-skills · ★ 0 · API & Backend · score 70
Install: claude install-skill brightstone111/web-scraping-skills
# Logged-in CDP + response-interception scraper Most scraping breaks because it fights the site: it re-logs-in with brittle form automation, parses HTML that changes weekly, and fires requests fast enough to trip bot defenses. This skill teaches a calmer approach that survives far longer: **Attach to a browser you already control, and read what it's already saying.** You launch your normal Chrome (with your real, human-established session), Playwright connects to it over CDP without opening a new browser, and you simply *listen* to the JSON the site's own frontend fetches as you navigate. No login automation, no HTML parsing, no synthetic API calls the server can fingerprint. ## Authorized use only This technique is for data you have a legitimate right to collect: your own accounts, sites whose Terms of Service permit it, APIs you're licensed for, or data you've been explicitly authorized to gather. Before scraping a third-party service, check its ToS and `robots.txt`, and keep request volume low enough that you never degrade the service for others. Automated collection can also violate a platform's ToS and get the *account* banned even when the IP is fine — so use a throwaway/dedicated account for anything against a service you don't own, never a personal or important one. When in doubt, prefer an official/paid API. ## Why response interception beats DOM scraping A modern web app renders from JSON it fetches over `fetch`/XHR — often a GraphQL endpoint. That JSON is ric