farzaa/clicky

hit this exact problem building a screen-aware voice agent on macOS. started with one provider, added a backend abstraction six months in. the transport layer isn't the hard part, the streaming semantics are what bite you. Claude sends tool_use deltas, OpenAI chunks function_call differently, and tool schemas aren't portable without a translation shim. POINT-style coord parsing is also where providers diverge most, Claude is consistent on format while GPT drifts without a tight system prompt. we kept the default direct path and treated gateway mode as an alternate transport behind a flag, otherwise every backend change turns into a regression hunt across the overlay layer. +1 on not coupling it to one specific gateway.

our macOS implementation of accessibility-based element targeting alongside coordinate control: https://github.com/mediar-ai/mcp-server-macos-use/blob/main/Sources/MCPServer/main.swift

and the cross-platform element abstraction that handles UIA on Windows: https://github.com/mediar-ai/terminator/blob/main/crates/terminator/src/element.rs

nice work on the Windows port. the HIPAA mode with local-only Whisper + SAPI is a solid differentiator.

one thing from building push-to-talk with vision on macOS: the screenshot capture timing matters more than you'd expect. if you capture the screenshot when the user starts speaking, by the time they finish their question (2-5 seconds later) the screen might have changed. we ended up capturing at push-to-talk release rather than press, and optionally a second capture mid-conversation if the user references something visual. the screen context at speech-end is almost always more relevant than at speech-start.

for the cursor overlay with POINT tags, have you considered using accessibility APIs alongside the coordinate-based approach? on Windows, UIA gives you element names and roles which makes the model's instructions more reliable than pixel coordinates. 'click the Submit button' is more robust than 'click at 450,320' especially across different screen resolutions.

the best solution would be to rely on [OpenResponses](https://www.openresponses.org) as the AI gateway endpoint.

I would like to see this as well but I don't think this should be restricted to OpenClaw as OpenClaw exposes both OpenAI and [OpenResponses](https://www.openresponses.org) endpoints. the later ones should be preferred for the generic implementation.

finally, please can you add Openrouter support so we can get rid of Anthropic ? Also please can you add
a text chatbot optional interaction (useful to take note of long replies) ?

Did you try it?

Really cool project, and I mean this in a friendly heads-up way, not a drive-by dunk.

A bit more detail on why I called these out:

- The Worker is doing the right job in principle by keeping secrets out of the binary, but without any auth or rate limiting on the Worker itself, anyone who gets the workers.dev URL can proxy requests through your Anthropic, ElevenLabs, and AssemblyAI accounts.
- The PostHog events currently look like they can include full user transcripts and full AI responses. Since the app also looks at screenshots, that creates a decent chance of collecting sensitive user or work content in analytics.
- The OpenAI transcription fallback appears to rely on an API key embedded in the app bundle, which weakens the otherwise solid keys-stay-server-side architecture.

The good news is these feel fixable without rethinking the product:

1. Add a thin auth layer or signed request scheme in front of the Worker.
2. Redact analytics down to counts, success/failure, and timing ...

You can have a look on this repo then
https://github.com/farzaa/clicky/issues/26
https://github.com/tekram/clicky-windows

@farzaa consider closing this issue

I've made a PR to use LMstudio has a free LLM provider, consider to use gemma-4-e4b, I've been running this on a Mac M1 pro

https://github.com/farzaa/clicky/pull/28

I made a pr which replaces all apis with native tools (Siri voice) and agent sdk so you can use your Claude subscription if you want to :)

https://github.com/farzaa/clicky/pull/29

in workers .dev.env add your api keys, it has paid elevenlabs key so tell claude to set free elevenlabs voice id instead unless you have paid elevenlabs version

finally I was looking for this

I have it working and all it requires anthropic key. will share screenshots soon