web-search

# Web Search & Scrape You have three tools for web access. Use them in combination based on what the task needs. ## The Stack ### SearXNG — Search Engine Local meta-search aggregating 25+ engines (Google, Bing, DuckDuckGo, Brave, etc). No tracking, no rate limits, JSON API. ```bash # Basic search curl -s "http://localhost:8888/search?q=QUERY&format=json" | python3 -c " import json, sys data = json.load(sys.stdin) for r in data.get('results', [])[:10]: print(r.get('title', '')) print(r.get('url', '')) print(r.get('content', '')[:200]) print() " ``` **Category search** — append `&categories=` with: `general`, `news`, `images`, `files`, `science`, `it`, `music`, `videos` ```bash # News search curl -s "http://localhost:8888/search?q=QUERY&format=json&categories=news" # Multiple categories curl -s "http://localhost:8888/search?q=QUERY&format=json&categories=science,it" ``` **Pagination** — append `&pageno=2` (or 3, 4, etc) for more results. ### Lightpanda — Fast Headless Fetch Built in Zig. 10x faster than Chrome, tiny memory footprint. Use this as the default for fetching page content. ```bash # Fetch as markdown (best for reading/summarizing) lightpanda fetch --dump markdown https://example.com # Fetch as HTML (when you need structure) lightpanda fetch --dump html https://example.com # Semantic tree (useful for understanding page layout) lightpanda fetch --dump semantic_tree https://example.com # Strip unnecessary elements lightpanda fetch --dump mar...