screen-visionlisted

# screen-vision > Governing principle (full text in the repo's `PHILOSOPHY.md`): **read the accessibility tree first, > use vision only to fill its gaps, and never act on a coordinate you cannot verify.** Structured UIA > data is exact and model-free; OCR/vision is the fallback, not the default. ## When to use / when to stop Use this when an agent needs to *see and act on the desktop*, capture the screen, get a structured list of buttons/text/inputs with **physical-pixel** coordinates, optionally click one. - Native apps, Win32/WinUI, Electron, games, remote-desktop, any non-browser window → **here**. - A web page's DOM/elements → **route to Playwright** (it reads the live DOM directly; this tool would only OCR pixels). "beyond the browser" in the description is the boundary. - Generating pixel art / editing an image file → not this skill (`pixel-art` / image tools). ## Workflow (thin) 1. **Probe once**, `python scripts/probe.py`. It reports DPI awareness, monitors, and which backends (UIA / OCR / annotate) are available, so you know what the next step can deliver. 2. **Capture (read-only, the default)**, `python scripts/capture.py --target ...`. Writes `screen.png`, an optional Set-of-Mark `annotated.png`, and `elements.json` (full), and prints a compact JSON summary + artifact paths to stdout. Read the summary to pick an element by **id**. 3. **Decide from JSON, confirm layout from the annotated PNG**, the JSON carries `center`, `rect`, `label`, `clickab