html-extraction-patternslisted

# HTML Extraction Patterns Most scrapers break for the same reason: they were written against the page that existed on the day, using selectors tied to how it happened to look. Six weeks later a designer changes a class name and the extraction silently returns nothing, or worse, returns the wrong thing. The fix is choosing selectors by how stable they are, not by what the browser inspector offers when you right-click. ## Look for structured data first Before writing a single selector, check whether the page already publishes what you want. A surprising share of the web does. ```bash # JSON-LD is often the entire record, cleanly typed curl -s https://example.com/product/123 \ | grep -oPz '(?s)<script type="application/ld\+json">.*?</script>' \ | sed -e 's/<[^>]*>//g' # Open Graph and meta tags curl -s https://example.com | grep -oE '<meta[^>]*(og:|twitter:|name="description")[^>]*>' # RSS, Atom and sitemaps: the site telling you its own structure curl -s https://example.com/sitemap.xml | grep -oP '(?<=<loc>)[^<]+' | head ``` **Then check for a JSON API behind the page.** Many listing pages are rendered from an internal endpoint the front end calls. Finding it turns a fragile HTML scrape into a clean, paginated, typed data source. Look in the network requests, or for a `__NEXT_DATA__` or similar hydration blob: ```bash curl -s https://example.com/listings | grep -oP '(?<=<script id="__NEXT_DATA__" type="application/json">).*?(?=</script>)' | head -c 2000 ``` This