Best architecture for integrating Python deep learning prototype into C++ production pipeline?

Not sure this is exactly what you're looking for, but if you want to interact between C++ and python this might be an interesting video to watch anyway : C++ ♥ Python - Alex Dathskovsky - CppCon 2025

If you want to use a very loose coupling between the Python and C++ part, you could try the following:

• Have the Python code run in its own background application.
• Establish or adopt a data sharing protocol between the C++ side and the Python side (who provides/expects which data, in which format?).

You could e.g. share C structs over a socket connection, but there are many more options for inter-process communication.

The main benefit that I see in contrast to a more tight integration: Only the communication interface (connection type and types of shared data) needs to be relatively stable, otherwise the C++ part should not care at all about who is providing the data (meaning the prototypes could be swapped out easily). With a socket connection, the Python code could even run on a different machine.

The main downside that I see: data transfer efficiency might not be optimal over such a connection; you may want to try whether or not it suffices your needs for prototyping. We used this setup in a company that I worked with some years ago. For our needs, it worked quite well – so good in fact, that we used the same concept in production.