computer-use

This skill provides the computer_use_* action tools for controlling a connected desktop. CU tools run through the main agent loop via HostCuProxy. The skill is internally preactivated for conversations with a connected desktop client. Tools in this skill are proxy tools. Execution is forwarded to a connected desktop client or handled by the assistant's virtual desktop. ## Observations Every computer-use step returns the accessibility tree. Every action also returns a screenshot taken after it ran (one at the end of a `computer_use_sequence`), so check it to see what your action did. An observation comes with a screenshot on a desktop's first look, when it is window-scoped, or when you pass `include_screenshot: true`. Ask for one whenever the tree is not enough to act on: a canvas, a game, a custom-drawn view, few or unlabeled controls, or a layout question. The tree is walked to a limited depth to keep steps fast, and says when it was cut off. If the element you need is not in it, call `computer_use_observe` with `full_tree: true`. ## Scripting apps (macOS) Reach for `computer_use_run_applescript` first when an app can be driven by script. It does not take the cursor. **Try the app's own scripting dictionary before anything else.** Ask it for the state you want, not for the clicks that would produce that state. Finder, Mail, Music, Notes, Safari, Terminal and many third-party apps have one, and where there is a dictionary the whole task is usually a sentence with no w...