app-control
FeaturedDrive a specific named macOS app via raw input bypassing the Accessibility tree
AI & Automation 954 stars
145 forks Updated today MIT
Install
Quality Score: 92/100
Stars 20%
Recency 20%
Frontmatter 20%
Documentation 15%
Issue Health 10%
License 10%
Description 5%
Skill Content
This skill exposes the `app_control_*` proxy tools for driving a single
named macOS application via raw input — keyboard, mouse, screenshot — that
bypasses the system Accessibility tree. Use it only when explicitly directed
to a specific app where the AX tree is unhelpful (emulators, games, OpenGL
canvases, custom-rendered Electron apps). For general macOS UI navigation
prefer the `computer-use` skill.
Tools in this skill are proxy tools — execution is forwarded to the connected
macOS client, never handled locally by the assistant.
## Cadence
Take 2-3 actions per turn, then yield with a short narration so the user can
interject. Do not chain long sequences without surfacing what you are doing.
## Always observe before acting
Call `app_control_observe` before your first input action whenever the screen
state matters (e.g. you need to know what is on screen, where a UI element is,
or whether the app is even running). Re-observe after actions that may have
moved the window or changed visibility.
`observe` waits a short settle delay (default ~200ms) before capturing so the
target app and the WindowServer can flush pending input and composite a fresh
frame. If the captured screenshot looks one input behind the latest state
(common with emulators or other slow-feedback apps), pass a larger
`settle_ms`. For static UIs where you just want a quick snapshot, pass
`settle_ms: 0` to skip the wait.
## Input choice
- Prefer `app_control_sequence` over multiple back-to-back `app_con...
Details
- Author
- vellum-ai
- Repository
- vellum-ai/vellum-assistant
- Created
- 5 months ago
- Last Updated
- today
- Language
- TypeScript
- License
- MIT
Integrates with
Similar Skills
Semantically similar based on skill content — not just same category
AI & Automation Listed
computer-use
Drive native macOS apps: click, type, screenshot windows without stealing focus. Use for desktop apps (Slack, Notes, Xcode), macOS workflow automation, or native window screenshots. macOS-only.
20 Updated yesterday
tomcounsell AI & Automation Solid
computer-use
Control the macOS desktop
954 Updated today
vellum-ai AI & Automation Listed
peekaboo
macOS UI automation: capture screenshots, click elements, type text, manage apps/windows/menus. Full desktop control via peekaboo CLI.
1 Updated yesterday
yourkenike