fetch-protectedlisted

# fetch-protected — read bot-walled pages with a real browser Plain HTTP clients (`curl`, `WebFetch`) get a challenge page instead of content on Cloudflare/DataDome/etc. TRAWL solves the challenge in a **real Camoufox Firefox context** (fingerprint-spoofed at the engine level, not JS patches — so it reads as a human browser), then returns the solved HTML. It's FlareSolverr-v2-compatible and caches solved sessions in Redis, so repeat hits to the same domain come back in ~500ms. ## When to use vs. not - **Use** when a fetch returns: `403`, `503`, "Just a moment…", "Checking your browser", "Enable JavaScript and cookies", a Turnstile/hCaptcha/reCAPTCHA widget, or a suspiciously empty body — i.e. a **bot challenge**, not a login. - **Don't use** for a real **auth gate** (Reddit "log in", paywalls, member areas). A solver passes bot checks; it does not have your account. For those, use a real logged-in session (OAuth token / persistent profile) — see `tools/web.md`. - **Escalate to it** only after a plain `WebFetch` fails. Plain HTTP is free and instant; the browser tier costs seconds. TRAWL itself does this internally (4 tiers: plain HTTP → cached session → live browser → residential proxy), so a single call is already cost-tiered. ## How (always via the helper — never hand-roll the JSON) Config: set `TRAWL_URL` in `.env` (e.g. `http://localhost:8191` or your homelab URL). ```bash tools/trawl.sh health # is a solver reachable? tool