Personal blog of Stein Magnus Jodal Zola 2011-03-20T00:00:00+00:00 https://jodal.dev/tags/multiprocessing/atom.xml 2011-03-20T00:00:00+00:00 2011-03-20T00:00:00+00:00 Unknown https://jodal.dev/2011/03/20/multiprocessing-connection-is-not-free/

<p>I’ve <a href="https://jodal.dev/2011/03/06/pickling-multiprocessing-connection-objects/">previously written</a> about how to wrap <code>multiprocessing.Connection</code> objects in a class that makes <code>Connection</code> objects picklable, and thus transferable over another <code>Connection</code>. That approach is still valid, but don’t use it–or <code>Connection</code> objects at all–for lots of use-once-and-throw-away connections.</p>

2011-03-06T00:00:00+00:00 2011-03-06T00:00:00+00:00 Unknown https://jodal.dev/2011/03/06/pickling-multiprocessing-connection-objects/

<p>For safe message-based communication between threads and processes in Python, I tend to use <a rel="external" href="http://docs.python.org/library/multiprocessing.html">multiprocessing</a>’s <code>Queue</code> and <code>Pipe</code>. A pattern often seen is using a queue for sending messages from multiple producers to a single consumer.</p> <p>When a producer wants a response to its message, I create a <code>Pipe</code> and piggy-back one end of the <code>Pipe</code> (a <code>Connection</code> object) to the message. I use Python dicts as messages, and use the string “reply_to” as the dictionary key for the connection objects.</p> <p>When the queue consumer processes a message, it doesn’t know who the sender is or how to reach him. Though, if the message has an attached <code>Connection</code> object, the consumer can–almost magically–respond to the sender, across thread and process boundaries.</p> <p>All good? Nope.</p>