extraction-validationlisted
Install: claude install-skill manypicom/web-data-skills
# Extraction Validation
A scraper that returns nothing gets noticed within the hour. A scraper that returns the page header 5,000 times, or two thirds of the records, or last month's prices, gets noticed when somebody acts on the data.
**Silent wrong output is the characteristic failure of extraction**, and no amount of careful selector writing prevents it. Validation is what catches it, and it has to run every time — not once when the scraper was built.
## The four layers
Run all of them. Each catches a different class of failure and the cheap ones catch most of it.
```
1. Schema Types, required fields, formats Cheap, catches structural breaks
2. Plausibility Ranges, lengths, patterns Cheap, catches selector drift
3. Distribution Compare this run to the last Cheap, catches partial breaks
4. Manual 20 records against their source_url Costly, catches everything else
```
Layer 3 is the one people skip, and it's the one that catches the most damaging failure: the scraper that still works for 70% of pages.
## Layer 1: Schema
Enforce the declared shape. See `schema-design-for-scraped-data`.
```python
from pydantic import ValidationError
ok, bad = [], []
for row in rows:
try:
ok.append(CompanyRecord(**row))
except ValidationError as e:
bad.append((row.get("source_url"), e.errors()))
print(f"{len(ok)} valid, {len(bad)} invalid")
for url, errs in bad[:10]:
print(url, [(x["loc"], x["msg"]) for x in