Precise UI interaction for AI agents on iOS, specifically tapping switch controls by reference.
Raw Developer Origin & Technical Request
GitHub Issue
Nov 10, 2026
### What happened, or what would make Pinky better?
On iOS, `phone_tap` with the `ref` of a switch row taps the centre of the row instead of the switch control. A connection switch at the top of an app's home page is reported as:
```text
[9] switch "node name" @645,465 checked
```
Steps:
1. Call `phone_tap { ref: 9, intent: "Turn the connection off" }`.
2. The call returns `settled`, but the switch stays on. Instead the long list behind the switch scrolls to a much lower position. `phone_observe` then shows rows that were previously off-screen.
3. Call `phone_tap { x: 1090, y: 465, intent: "Turn the connection off" }` — a point inside the switch's thumb on the right of the row. The element changes to an unchecked switch with a disconnected label.
4. Tap the same point again. The element changes back to a checked switch with the node name.
**Expected:** tapping a `switch` element by `ref` changes the switch, as it does for a button. If the accessibility frame spans the whole row, Pinky should target the switch control within that frame, or report that the tap did not activate it. Returning `settled` without a state change makes an agent believe a disconnect/reconnect happened when it did not.
**Setup:** iPhone 15 Plus over Wi-Fi, Pinky 2026.1110.2 for Windows, MCP `phone_*` tools. Screen size 1290x2796; the switch is centred at y=465 and its thumb is around x=1090. The row's reported centre x=645 is over the label, not the control.
This is reproducible on the same switc...
Developer Debate & Comments
No active discussions extracted for this entry yet.
Adjacent Repository Pain Points
Other highly discussed features and pain points extracted from DIYgod/pinky.
点击全部停止无效
Frequently Asked Questions
Market intelligence mapped to Precise UI interaction for AI agents on iOS, specifically tapping switch controls by reference..
What is the technical positioning of Precise UI interaction for AI agents on iOS, specifically tapping switch controls by reference.?
What is the general sentiment around Precise UI interaction for AI agents on iOS, specifically tapping switch controls by reference.?
What architecture is tied to Precise UI interaction for AI agents on iOS, specifically tapping switch controls by reference.?
Engagement Signals
Cross-Market Term Frequency
Counts stored text matches for foundational terms like ref and iOS in analysed public discussions. These counts do not measure adoption.
Stored text matches, not adoption or market-share measurements.
- ref: 120 records
- iOS: 67 records
- Wi-Fi: 6 records
- settled: 3 records
- phone_tap: 2 records
- iPhone 15 Plus: 2 records
- switch control: 1 records
- row centre: 1 records
- accessibility frame: 1 records
- phone_observe: 1 records
- x: 1090, y: 465: 1 records
- unchecked switch: 1 records
- checked switch: 1 records
- node name: 1 records
- disconnected label: 1 records
- Pinky 2026.1110.2 for Windows: 1 records
- MCP phone_* tools: 1 records
- screen size 1290x2796: 1 records
SaaS Metrics