wsb-data-pipelinelisted

Scrape r/wallstreetbets posts for specific stock/forex tickers (TSLA, AAPL, XAUUSD, NVDA, GME) over historical date ranges and export Post Title, Date, Upvotes, Number of Comments, and Ticker Mentions to an Excel workbook. Use when the user asks to collect, update, or re-run WSB Reddit data for the herd-behavior research project, or asks about ticker date ranges, the data source, or output columns for that dataset.
Faaz17/Agents-and-Skills · ★ 1 · Data & Documents · score 74

Install: claude install-skill Faaz17/Agents-and-Skills

# WSB Data Pipeline Collects historical r/wallstreetbets post data per ticker into `output/wsb_ticker_data.xlsx` (one sheet per ticker) for the herd-behavior sentiment research (see `Research Points.docx`). ## Data source Reddit's live API/search does not return multi-year-old posts reliably. This pipeline uses the **Arctic Shift API** (`https://arctic-shift.photon-reddit.com/api/posts/search`), a free public mirror of the Pushshift historical Reddit archive, instead. No Reddit credentials are required. ## Running the pipeline Always run the bundled script — do not hand-write scraping/Excel code, the script already handles pagination, rate-limit backoff, caching, and deduping. ```bash cd "<project root>" .venv/Scripts/python.exe scripts/scrape_wsb.py # all 5 tickers, full configured ranges .venv/Scripts/python.exe scripts/scrape_wsb.py --ticker GME # one ticker only .venv/Scripts/python.exe scripts/scrape_wsb.py --ticker TSLA --after 2024-01-01 --before 2024-02-01 # custom range .venv/Scripts/python.exe scripts/scrape_wsb.py --no-cache # force re-fetch, ignore cached API pages ``` On Windows PowerShell use `.\.venv\Scripts\python.exe` instead of the forward-slash form. Each run merges into the existing `output/wsb_ticker_data.xlsx` (sheets for tickers not re-run are preserved), so a single ticker can be refreshed without re-scraping the rest. ## Configured ticker ranges (in `scripts/scrape_wsb.py`, `TICKERS` dict) | Ticker | Aliases search