app-control

Solid

Drive a specific named macOS app via raw input bypassing the Accessibility tree

AI & Automation 648 stars 94 forks Updated today MIT

Install

View on GitHub

Quality Score: 91/100

Stars 20%
94
Recency 20%
100
Frontmatter 20%
70
Documentation 15%
100
Issue Health 10%
50
License 10%
100
Description 5%
100

Skill Content

This skill exposes the `app_control_*` proxy tools for driving a single named macOS application via raw input — keyboard, mouse, screenshot — that bypasses the system Accessibility tree. Use it only when explicitly directed to a specific app where the AX tree is unhelpful (emulators, games, OpenGL canvases, custom-rendered Electron apps). For general macOS UI navigation prefer the `computer-use` skill. Tools in this skill are proxy tools — execution is forwarded to the connected macOS client, never handled locally by the assistant. ## Cadence Take 2-3 actions per turn, then yield with a short narration so the user can interject. Do not chain long sequences without surfacing what you are doing. ## Always observe before acting Call `app_control_observe` before your first input action whenever the screen state matters (e.g. you need to know what is on screen, where a UI element is, or whether the app is even running). Re-observe after actions that may have moved the window or changed visibility. `observe` waits a short settle delay (default ~200ms) before capturing so the target app and the WindowServer can flush pending input and composite a fresh frame. If the captured screenshot looks one input behind the latest state (common with emulators or other slow-feedback apps), pass a larger `settle_ms`. For static UIs where you just want a quick snapshot, pass `settle_ms: 0` to skip the wait. ## Input choice - Prefer `app_control_sequence` over multiple back-to-back `app_con...

Details

Author
vellum-ai
Repository
vellum-ai/vellum-assistant
Created
4 months ago
Last Updated
today
Language
TypeScript
License
MIT

Integrates with

Similar Skills

Semantically similar based on skill content — not just same category