farzaa/clicky

our macOS implementation of accessibility-based element targeting alongside coordinate control: https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift

and the cross-platform element abstraction that handles UIA on Windows: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

nice work on the Windows port. the HIPAA mode with local-only Whisper + SAPI is a solid differentiator.

one thing from building push-to-talk with vision on macOS: the screenshot capture timing matters more than you'd expect. if you capture the screenshot when the user starts speaking, by the time they finish their question (2-5 seconds later) the screen might have changed. we ended up capturing at push-to-talk release rather than press, and optionally a second capture mid-conversation if the user references something visual. the screen context at speech-end is almost always more relevant than at speech-start.

for the cursor overlay with POINT tags, have you considered using accessibility APIs alongside the coordinate-based approach? on Windows, UIA gives you element names and roles which makes the model's instructions more reliable than pixel coordinates. 'click the Submit button' is more robust than 'click at 450,320' especially across different screen resolutions.

the best solution would be to rely on [OpenResponses](https://www.openresponses.org) as the AI gateway endpoint.

finally, please can you add Openrouter support so we can get rid of Anthropic ? Also please can you add
a text chatbot optional interaction (useful to take note of long replies) ?

Did you try it?

@farzaa consider closing this issue

I've made a PR to use LMstudio has a free LLM provider, consider to use gemma-4-e4b, I've been running this on a Mac M1 pro

https://github.com/farzaa/clicky/pull/28

I made a pr which replaces all apis with native tools (Siri voice) and agent sdk so you can use your Claude subscription if you want to :)

https://github.com/farzaa/clicky/pull/29

in workers .dev.env add your api keys, it has paid elevenlabs key so tell claude to set free elevenlabs voice id instead unless you have paid elevenlabs version

finally I was looking for this

I have it working and all it requires anthropic key. will share screenshots soon

https://github.com/tekram/clicky-windows

want to collborate?

+1

> I love the concept but I appreciate version for windows. if anyone is vibecoding it right now, please, add a link to this issues.

I am building cross platform version for Windows and Linux

Implemented in https://github.com/farzaa/clicky/pull/20 - replaced the empty Settings scene with a short message pointing users to the menu bar icon.