webcrawler-deep-crawl

# Website Deep Crawl > Input: one or more start URLs (+ optional scope, depth, page-count, globs, removal selectors). Output: per-page records `{url, crawl, metadata, text, markdown, html, outboundLinks}` for every page reached within scope. ## Language All process output to user (progress updates, process notifications) follows the user's language. ## Objective From a small set of start URLs, breadth-first crawl every reachable in-scope page, strip boilerplate (navigation, header, footer, cookie banners, etc.), and emit per-page LLM-ready content (text / markdown / HTML) plus structured metadata — suitable for feeding RAG pipelines, vector databases, or chatbot knowledge bases. ## Prerequisites - One or more start URLs are provided by the caller. - Target pages are publicly reachable, OR the running browser is already logged in for any pages behind authentication. - A working directory is available for writing per-page JSON records and the crawl state file. ## Pre-execution Checks ### 1. Tool Readiness If browser-act has been confirmed available in the current session → skip this step. Invoke `browser-act` via Skill tool to load usage. If installation or configuration issues arise, follow its guidance to resolve then retry. ### 2. Login Verification (when prerequisites include login requirement) If login status for the target site has been confirmed in the current session → skip this step. Otherwise: open the target site and observe the page login status: - Log...