Error group argument must be None for now in multiprocessing.pool

2024/11/16 11:37:05

Below is my python script.

import multiprocessing
# We must import this explicitly, it is not imported by the top-level
# multiprocessing module.
import multiprocessing.pool
import timefrom random import randintclass NoDaemonProcess(multiprocessing.Process):# make 'daemon' attribute always return Falsedef _get_daemon(self):return Falsedef _set_daemon(self, value):passdaemon = property(_get_daemon, _set_daemon)# We sub-class multiprocessing.pool.Pool instead of multiprocessing.Pool
# because the latter is only a wrapper function, not a proper class.
class MyPool(multiprocessing.pool.Pool):Process = NoDaemonProcessdef sleepawhile(t):print("Sleeping %i seconds..." % t)time.sleep(t)return tdef work(num_procs):print("Creating %i (daemon) workers and jobs in child." % num_procs)pool = multiprocessing.Pool(num_procs)result = pool.map(sleepawhile,[randint(1, 5) for x in range(num_procs)])# The following is not really needed, since the (daemon) workers of the# child's pool are killed when the child is terminated, but it's good# practice to cleanup after ourselves anyway.pool.close()pool.join()return resultdef test():print("Creating 5 (non-daemon) workers and jobs in main process.")pool = MyPool(20)result = pool.map(work, [randint(1, 5) for x in range(5)])pool.close()pool.join()print(result)if __name__ == '__main__':test()

This is running in ubuntu server and i'm using python 3.6.7

I had this working properly after apt-get upgrade Im getting error as

group argument must be None for now

What might be the error that I'm facing. Should i change the python version. Should I roll back the changes after upgrading.

EDIT 1

Stacktrace exception:-

Traceback (most recent call last):File "/src/mainapp.py", line 104, in bulkfunp = MyPool(20)File "/usr/lib/python3.6/multiprocessing/pool.py", line 175, in __init__self._repopulate_pool()File "/usr/lib/python3.6/multiprocessing/pool.py", line 236, in _repopulate_poolself._wrap_exception)File "/usr/lib/python3.6/multiprocessing/pool.py", line 250, in _repopulate_pool_staticwrap_exception)File "/usr/lib/python3.6/multiprocessing/process.py", line 73, in __init__assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now

EDIT 2

The code works for python2.7, python3.5 But if i run with python 3.6.7 i got the error as below.

Creating 5 (non-daemon) workers and jobs in main process.
Traceback (most recent call last):File "multi.py", line 52, in <module>test()File "multi.py", line 43, in testpool = MyPool(5)File "/usr/lib/python3.6/multiprocessing/pool.py", line 175, in __init__self._repopulate_pool()File "/usr/lib/python3.6/multiprocessing/pool.py", line 236, in _repopulate_poolself._wrap_exception)File "/usr/lib/python3.6/multiprocessing/pool.py", line 250, in _repopulate_pool_staticwrap_exception)File "/usr/lib/python3.6/multiprocessing/process.py", line 73, in __init__assert group is None, 'group argument must be None for now'
AssertionError: group argument must be None for now
Answer

I came across to this issue while upgrading Travis distribution from 14.04 to 16.04 and python 3.6 started to fail. I have found a solution to this problem as it was a fix to another package - FIX: Python 2.7-3.7.1 compatible NonDaemonPool

class NonDaemonPool(multiprocessing.pool.Pool):def Process(self, *args, **kwds):proc = super(NonDaemonPool, self).Process(*args, **kwds)class NonDaemonProcess(proc.__class__):"""Monkey-patch process to ensure it is never daemonized"""@propertydef daemon(self):return False@daemon.setterdef daemon(self, val):passproc.__class__ = NonDaemonProcessreturn proc
https://en.xdnf.cn/q/71675.html

Related Q&A

Making the diamond square fractal algorithm infinite

Im trying to generate an infinite map, as such. Im doing this in Python, and I cant get the noise libraries to correctly work (they dont seem to ever find my VS2010, and doing it in raw Python would be…

How do I generate coverage xml report for a single package?

Im using nose and coverage to generate coverage reports. I only have one package right now, ae, so I specify to only cover that: nosetests -w tests/unit --with-xunit --with-coverage --cover-package=aeA…

Asynchronous URLfetch when we dont care about the result? [Python]

In some code Im writing for GAE I need to periodically perform a GET on a URL on another system, in essence pinging it and Im not terribly concerned if the request fails, times out or succeeds.As I bas…

Python: How to fill out form all at once with splinter/Browser?

Currently, I’m filling out the form on a site with the following:browser.fill(‘form[firstname]’, ‘Mabel’) browser.fill(‘form[email]’, ‘[email protected]’) browser.select(‘form[color]’, ‘yel…

Dump elementtree into xml file

I created an xml tree with something like thistop = Element(top) child = SubElement(top, child) child.text = some texthow do I dump it into an XML file? I tried top.write(filename), but the method doe…

Crash reporting in Python

Is there a crash reporting framework that can be used for pure Python Tkinter applications? Ideally, it should work cross-platform.Practically speaking, this is more of exception reporting since the P…

Python SocketServer

How can I call shutdown() in a SocketServer after receiving a certain message "exit"? As I know, the call to serve_forever() will block the server.Thanks!

Loading JSON file in BigQuery using Google BigQuery Client API

Is there a way to load a JSON file from local file system to BigQuery using Google BigQuery Client API?All the options I found are:1- Streaming the records one by one.2- Loading JSON data from GCS.3- …

How to extract tables from a pdf with PDFMiner?

I am trying to extract information from some tables in a pdf document. Consider the input:Title 1 some text some text some text some text some text some text some text some text some text some textTabl…

Draw Box-Plot with matplotlib

Is it possible to plot this kind of chart with matplotlib?