Multiprocessing Pool - how to cancel all running processes if one returns the desired result?

2024/9/20 5:41:25

Given the following Python code:

import multiprocessingdef unique(somelist):return len(set(somelist)) == len(somelist)if __name__ == '__main__':somelist = [[1,2,3,4,5,6,7,8,9,10,11,12,13,2], [1,2,3,4,5], [1,2,3,4,5,6,7,8,9,1], [0,1,5,1]]pool = multiprocessing.Pool()reslist = pool.map(unique, somelist)pool.close()pool.join()print "Done!"print reslist

Now imagine, that the lists with integers in this toy example are extremely long, and what I'd like to achieve here is the following: if one of the lists in somelist returns True, kill all running processes.

This leads to two questions (and probably more which I haven't come up with):

  • How can I "read" or "listen" from a finished process the result, while other processes are running? If e.g. a process is dealing with [1,2,3,4,5] from somelist, and is finished before all other processes, how can I read out the result from that process in this very moment?

  • Given the case that it is possible to "read" out the result of a finished process while other are running: how can I use this result as a condition to terminate all other running processes?

    e.g. If one process has finished and returned True, how I can use this as a condition to terminate all other (still) running processes?

Answer

Use pool.imap_unordered to view the results in any order they come up.

reslist = pool.imap_unordered(unique, somelist)
pool.close()
for res in reslist:if res:  # or set other condition herepool.terminate()break
pool.join()

You can iterate over an imap reslist in your main process while the pool processes are still generating results.

https://en.xdnf.cn/q/72383.html

Related Q&A

Pandas: Product of specific columns

Finding the product of all columns in a dataframe is easy:df[Product] = df.product(axis=1)How can I specify which column names (not column numbers) to include in the product operation?From the help pa…

instagram.bind.InstagramClientError: Unable to parse response, not valid JSON

Made myself a simple Instagram client to make authenticated requests to their API. However, running it keeps throwing the following errorTraceback (most recent call last):File "request-ig-data.py&…

Using jinja to send data to Javascript

I have Python code, in which Im using jinja to send data to a template in Flask. I can access the code just find in HTML, but when I try displaying the data in Javascript, it doesnt work. For example, …

celery: Substantial drift from

I have quite a problem with celery on my distribted system. I have couple of machines among different localizations and Ive got a lot of warnings in my log files like:"Substantial drift from celer…

jinja2 link to static files

I am trying to understand how to create a link to static files in jinja2.Everything I look up relates to Flask whereas I am using just webapp2 at this stage.My main.py file looks as follows:import os i…

How to upgrade tensorflow with GPU on google colaboratory

Currently google colaboratory uses tensorflow 1.4.1. I want to upgrade it to 1.5.0 version. Each time when i executed !pip install --upgrade tensorflow command, notebook instance succesfully upgrades t…

How to print warnings and errors when using setuptools (pip)

I am using setuptools to package code such that it can be easily installed using cd project_name && pip install .During the setup process, I want to warn the user about pre-existing config file…

TypeError: not supported between instances of State and State PYTHON 3

I am trying to utilize a PriorityQueue from the queue class. However, im having issues putting custom objects into my PQ. I have implemented the __cmp__ function below:def __cmp__(self, other):return (…

Threading and information passing -- how to

To reframe from confusion i have edited the question:one.pyimport threading count = 5 dev = threading.Thread(name=dev, target=dev,args=(workQueue,count,)) dev.setDaemon(True) dev.start() workQueue = Qu…

How to rename a node with Python LXML?

How do I rename a node using LXML? Specifically, how to rename a parent node i.e. a <body> tag while preserving all the underlying structure? I am parsing using the lxml.html module but suppos…