download-gated-pdfs

# Download bot-gated PDFs via Wayback id_ ## Problem Many think-tank and publisher sites (taxpolicycenter.org, urban.org, SSRN delivery) serve an HTML bot-challenge page instead of the PDF to non-browser clients. A browser User-Agent header does not help. The downloaded "PDF" is actually HTML. ## Context / Trigger Conditions - `curl -o file.pdf <url>` succeeds but the file starts with `<!DOC` - pypdf raises `invalid pdf header: b'<!DOC'` or `PdfStreamError: Stream has ended unexpectedly` - Firecrawl `scrape` returns clean markdown for the same URL (its proxies get through), but Firecrawl does not return the binary — only parsed content ## Solution 1. Request the file through the Wayback Machine's raw-content (`id_`) endpoint, which serves the original archived binary without rewriting: ```sh curl -sL -A "Mozilla/5.0 ... Chrome/126.0 Safari/537.36" \ "https://web.archive.org/web/<YYYY>id_/<original-pdf-url>" -o out.pdf ``` `<YYYY>` is any year likely to have a snapshot (e.g. publication year); Wayback redirects to the nearest capture. The `id_` suffix after the timestamp is what requests the untouched original. 2. Verify the download with pypdf — a bot page fails immediately: ```python from pypdf import PdfReader r = PdfReader("out.pdf"); print(len(r.pages), "pages") ``` 3. If Wayback has no capture, fall back to: another mirror found via search (Exa/Firecrawl), or Firecrawl scrape for the parsed text when the binary is not stri...