Skills macpilot-ui-inspector

Name: macpilot-ui-inspector
Author: openclaw

Inspect and interact with macOS UI elements using MacPilot accessibility APIs. Find buttons, text fields, labels, and other elements by role, label, or position, then click, read, or modify them.

install

source · Clone the upstream repo

git clone https://github.com/openclaw/skills

Claude Code · Install into ~/.claude/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/openclaw/skills "$T" && mkdir -p ~/.claude/skills && cp -r "$T/skills/adhikjoshi/macpilot/skills/macpilot-ui-inspector" ~/.claude/skills/openclaw-skills-macpilot-ui-inspector && rm -rf "$T"

OpenClaw · Install into ~/.openclaw/skills/

T=$(mktemp -d) && git clone --depth=1 https://github.com/openclaw/skills "$T" && mkdir -p ~/.openclaw/skills && cp -r "$T/skills/adhikjoshi/macpilot/skills/macpilot-ui-inspector" ~/.openclaw/skills/openclaw-skills-macpilot-ui-inspector && rm -rf "$T"

manifest: skills/adhikjoshi/macpilot/skills/macpilot-ui-inspector/SKILL.md

MacPilot UI Inspector

Use MacPilot's accessibility (AX) commands to inspect, find, and interact with UI elements in any macOS application. This skill enables precise programmatic control of app interfaces.

When to Use

Use this skill when the user asks to:

Find a specific button, text field, checkbox, or UI element in an app
Read the value or state of a UI control
Click a button or interact with a control by its label
Inspect the full UI hierarchy/accessibility tree of an app
Set values in text fields or toggle checkboxes programmatically
Discover what keyboard shortcuts an app supports
Identify elements at specific screen coordinates

Commands

List UI Elements

macpilot ui list --app "Safari" --json                    # All elements
macpilot ui list --app "Safari" --role AXButton --json    # Only buttons
macpilot ui list --app "Safari" --depth 5 --json          # Deeper scan
macpilot ui list --app "Safari" --hierarchy --json        # With hierarchy

Find Elements by Text

macpilot ui find "Save" --app "TextEdit" --json           # Find by label
macpilot ui find "Save" --role AXButton --json            # Filter by role
macpilot ui find "Save" --exact --json                    # Exact match only

Find Element with Coordinates

macpilot ui find-text "Submit" --app "Safari" --json
# Returns: position {x, y}, size {w, h} - use for clicking

Click UI Element by Label

macpilot ui click "Save" --app "TextEdit" --json
macpilot ui click "Cancel" --role AXButton --json

Get/Set Values

macpilot ui get-value "Search" --app "Safari" --json          # Read field value
macpilot ui set-value "Search" "query text" --app "Safari"    # Set field value
macpilot ui set-value "Dark Mode" "1" --role AXCheckBox       # Toggle checkbox

Set Focus

macpilot ui set-focus "Search" --app "Safari" --json    # Focus an element

Scroll Within Elements

macpilot ui scroll "content" down 5 --app "Safari"     # Scroll element

Inspect Element Attributes

macpilot ui attributes "Save" --app "TextEdit" --json   # All AX attributes

Elements at Coordinates

macpilot ui elements-at 500 300 --json                  # What's at x=500 y=300
macpilot ui elements-at 500 300 --radius 50 --json      # Search wider area

Accessibility Tree

macpilot ui tree --app "Finder" --json                  # Full AX tree
macpilot ui tree --app "Finder" --depth 3 --json        # Limit depth

Keyboard Shortcuts

macpilot ui shortcuts --app "Safari" --json             # All shortcuts
macpilot ui shortcuts --app "Safari" --menu File --json # Menu-specific

Wait for Elements

macpilot wait element "Download Complete" --app "Safari" --timeout 30 --json

Common AX Roles

Role	Description
`AXButton`	Buttons (push, toggle)
`AXTextField`	Text input fields
`AXTextArea`	Multi-line text areas
`AXStaticText`	Labels and display text
`AXCheckBox`	Checkboxes and toggles
`AXRadioButton`	Radio buttons
`AXPopUpButton`	Dropdown menus
`AXComboBox`	Combo boxes
`AXTable`	Tables and lists
`AXRow`	Table/list rows
`AXMenuItem`	Menu items
`AXToolbar`	Toolbars
`AXScrollArea`	Scrollable regions
`AXWindow`	Windows
`AXSheet`	Modal sheets
`AXImage`	Images
`AXLink`	Hyperlinks
`AXGroup`	Generic container

Workflow Patterns

Pattern 1: Find and Click

# Always focus app first
macpilot app focus "Safari"
# Find the element to verify it exists
macpilot ui find "Downloads" --app "Safari" --role AXButton --json
# Click it
macpilot ui click "Downloads" --app "Safari"

Pattern 2: Read Form State

macpilot app focus "System Settings"
macpilot ui get-value "Computer Name" --app "System Settings" --json

Pattern 3: Fill a Form

macpilot app focus "MyApp"
macpilot ui set-value "Name" "John Doe" --app "MyApp"
macpilot ui set-value "Email" "john@example.com" --app "MyApp"
macpilot ui click "Submit" --app "MyApp"

Pattern 4: Explore Unknown UI

# Start with a broad scan
macpilot ui tree --app "SomeApp" --depth 2 --json
# Narrow down to specific elements
macpilot ui list --app "SomeApp" --role AXButton --json
# Inspect a specific element
macpilot ui attributes "Settings" --app "SomeApp" --json

Pattern 5: Click by Coordinates (Fallback)

# When label-based clicking fails, find coordinates first
macpilot ui find-text "Submit" --app "MyApp" --json
# Use returned x,y to click
macpilot click 450 320

Tips

Always call
```
macpilot app focus
```
before any UI interaction
Use
```
--json
```
output for reliable parsing of element properties
When
```
ui click
```
fails, fall back to
```
ui find-text
```
+
```
macpilot click x y
```
Use
```
ui tree --depth 2
```
first to understand app structure before deep scans
AX permission must be granted to MacPilot.app in System Settings
The
```
--role
```
filter significantly speeds up element searches in complex apps