Python multiprocessing pipe recv() doc unclear or did I miss anything?

2024/10/8 4:28:54

I have been learning how to use the Python multiprocessing module recently, and reading the official doc. In 16.6.1.2. Exchanging objects between processes there is a simple example about using pipe to exchange data.

And, in 16.6.2.4. Connection Objects, there is this statement, quoted "Raises EOFError if there is nothing left to receive and the other end was closed."

So, I revised the example as shown below. IMHO this should trigger an EOFError exception: nothing sent and the sending end is closed.

The revised code:

from multiprocessing import Process, Pipedef f(conn):#conn.send([42, None, 'hello'])conn.close()if __name__ == '__main__':parent_conn, child_conn = Pipe()p = Process(target=f, args=(child_conn,))p.start()#print parent_conn.recv()   # prints "[42, None, 'hello']"try:print parent_conn.recv()except EOFError:passp.join()

But, when I tried the revised example on my Ubuntu 11.04 machine, Python 2.7.2, the script hang.

If anyone can point out to me what I missed, I would be very appreciative.

Answer

When you start a new process with mp.Process, the child process inherits the pipes of the parent. When the child closes conn, the parent process still has child_conn open, so the reference count for the pipe file descriptor is still greater than 0, and so EOFError is not raised.

To get the EOFError, close the end of the pipe in both the parent and child processes:

import multiprocessing as mpdef foo_pipe(conn):conn.close()def pipe():conn = mp.Pipe()parent_conn, child_conn = connproc = mp.Process(target = foo_pipe, args = (child_conn, ))proc.start()child_conn.close()  # <-- Close the child_conn end in the main process too.try:print(parent_conn.recv())except EOFError as err:print('Got here')proc.join()if __name__=='__main__':pipe()
https://en.xdnf.cn/q/70161.html

Related Q&A

How would you unit test this SQLAlchemy Core query/function?

Im working on learning how to unit test properly. Given this function...def get_user_details(req_user_id):users = sa.Table(users, db.metadata, autoload=True)s = sa.select([users.c.username,users.c.favo…

Retrieve wall-time in Python using the standard library?

How can I retrieve wall-time in Python using the standard library?This question, and this question would suggest that something like clock_gettime(CLOCK_MONOTONIC_RAW) or /proc/uptime are most appropr…

NLTK: Package Errors? punkt and pickle?

Basically, I have no idea why Im getting this error. Just to have more than an image, here is a similar message in code format. As it is more recent, the answer of this thread has already been mentione…

Is there a bit-wise trick for checking the divisibility of a number by 2 or 3?

I am looking for a bit-wise test equivalent to (num%2) == 0 || (num%3) == 0.I can replace num%2 with num&1, but Im still stuck with num%3 and with the logical-or.This expression is also equivalent …

Check image urls using python-markdown

On a website Im creating Im using Python-Markdown to format news posts. To avoid issues with dead links and HTTP-content-on-HTTPS-page problems Im requiring editors to upload all images to the site and…

How to unittest command line arguments?

I am trying to supply command line arguments to Python unittest and facing some issues. I have searched on internet and found a way to supply arguments asunittest.main(argv=[myArg])The issue is this wo…

different foreground colors for each line in wxPython wxTextCtrl

I have a multilinewx.TextCtrl()object which I set its forground and Background colors for writing strings.I need to write different lines with different colors ,wx.TextCtrl.setForgroundcolor()changes a…

Access deprecated attribute validation_data in tf.keras.callbacks.Callback

I decided to switch from keras to tf.keras (as recommended here). Therefore I installed tf.__version__=2.0.0 and tf.keras.__version__=2.2.4-tf. In an older version of my code (using some older Tensorfl…

How to unpickle a file that has been hosted in a web URL in python

The normal way to pickle and unpickle an object is as follows:Pickle an object:import cloudpickle as cpcp.dump(objects, open("picklefile.pkl", wb))UnPickle an object: (load the pickled file):…

Control tick-labels from multi-level FactorRange

Ive got a three-level bokeh.models.FactorRange which I use to draw tick labels on a vbar-plot. The problem is that there are dozens of factors in total and the lowest-level labels get very cramped.I ca…