facebookincubator/muse-gadget-sdk

Following the suggestion above to use a TTS API of your choice, here is a working implementation in case it's useful to others: https://github.com/chucky1102/muse-gadget-sdk/tree/local-tts

- `CONFIG_MUSE_LOCAL_TTS`: for each reply, the firmware POSTs the text to an HTTP server you run and plays the MP3 it streams back, through the SDK's existing MP3 decoder and speaker path. It's the hook `start_tts()` describes. Captions follow the speech as before.
- A small server built on `edge-tts` is included, with setup steps and the protocol for swapping in another TTS: [tts_server/README.md](https://github.com/chucky1102/muse-gadget-sdk/blob/local-tts/esp32/tools/muse/tts_server/README.md)
- Speech starts a little over a second after the reply's text arrives.

One thing others may hit on ESP-IDF 6.0.1: `vStreamBufferDeleteWithCaps()` can free the buffer twice (espressif/esp-idf#18855). Here it showed up as an occasional reboot at the end of a reply. Creating the stream buffer with `xStreamBuf...

@Valentinetemi For me, the most important thing is to send pictures from the gadget to Muse.

I can't speak for the Op, but I can say that I would be interested in seeing either if not both implemented.

Hi! I’d be interested in working on this. By image messages, do you mean sending an image from a gadget to Muse, or receiving/displaying image messages from Muse on the gadget? I can take a look at implementing it once the expected flow is clarified.

@anantn Hey Please review my Pull request

We're still working on a robust voice pipeline for gadgets and hope to share more on that in the near future! In the short term, I'd recommend a TTS API of your choice for spoken responses. I'll leave this issue open so we can post updates when they are available.