Detect when multiprocessing queue is empty and closed

2024/10/14 7:16:14

Let's say I have two processes: a reader and a writer. How does the writer detect when the reader has finished writing values?

The multiprocessing module has a queue with a close method that seems custom-built for this purpose. But how do you detect when the queue has been closed?

This doesn't seem to work, as the writer never exits:

import multiprocessing as mpdef getter(q):while True:try:print(q.get())except Exception:breakdef putter(q):q.put(1)q.put(2)q.close()q = mp.Queue()
writer = mp.Process(target=putter, args=(q, ))
reader = mp.Process(target=getter, args=(q, ))
reader.start()
writer.start()writer.join()
reader.join()

Should the reader-writer pair use a sentinel value to signal end of writing? Then what's the point of having the close method?

EDIT: While this question asks about the Queue module (now queue), I am asking specifically about mp.Queue and what the correct use of the .close method is.

Answer

But how do you detect when the queue has been closed?

You don't. That is not the purpose of close. Calling close doesn't even guarantee that no more items will be added to the queue; the docs say

Indicate that no more data will be put on this queue by the current process.

close is intended to shut down the current process's feeder thread for that queue (or at least start shutting it down), not to communicate an end-of-queue to other processes.


If you want to signal that no more values will be written to the queue, use standard techniques like enqueueing a sentinel object, like you would with an ordinary queue.Queue.

https://en.xdnf.cn/q/69439.html

Related Q&A

imshow and histogram2d: cant get them to work

Im learning Python and this is my first question here. Ive read other topics related to the usage of imshow but didnt find anything useful. Sorry for my bad English.I have plotted a set of points here,…

3D Waterfall Plot with Colored Heights

Im trying to visualise a dataset in 3D which consists of a time series (along y) of x-z data, using Python and Matplotlib.Id like to create a plot like the one below (which was made in Python: http://a…

python xlsxwriter: Keep header in excel when adding a table

I have a panda dataframe that I write to a xslx file, and would like to add a table over that data. I would also like to keep the headers that I have already written, instead of adding them again. Is t…

Unexpected tokens in !DOCTYPE html in PyCharm Community Edition

I am new in using PyCharm but I am loving it gradually. I am getting a red underline on <!DOCTYPE html> and the error is "Unexpected Token".Why PyCharm shows it? I cant understand.

Indexing a numpy array with a numpy array of indexes [duplicate]

This question already has answers here:Indexing a numpy array with a list of tuples(2 answers)Index multidimensional array with index array(1 answer)Closed 5 years ago.I have a 3D numpy array data and …

Input redirection with python

I have the following program to test input redirection in Python.a = int(raw_input("Enter a number: ")) b = raw_input("Enter a string: ") print "number entered = ", a prin…

TypeError: can only concatenate str (not numpy.int64) to str

I want to print the variable based on the index number based on the following dataset:Here I used the following code:import pandas as pdairline = pd.read_csv("AIR-LINE.csv")pnr = input("…

Saving scatterplot animations with matplotlib produces blank video file

I am having a very similar problem to this questionbut the suggested solution doesnt work for me. I have set up an animated scatter plot using the matplotlib animation module. This works fine when it i…

How to group near-duplicate values in a pandas dataframe?

If there are duplicate values in a DataFrame pandas already provides functions to replace or drop duplicates. In many experimental datasets on the other hand one might have near duplicates. How can one…

python looping and creating new dataframe for each value of a column

I want to create a new dataframe for each unique value of station.I tried below which gives me only last station data updated in the dataframe = tai_new.itai[station].unique() has 500 values.for i in t…