Can someone explain why this code blocks and cannot complete?

I've followed a couple of examples for multiprocessing and I've writting some very similar code that does not get blocked. But, obviously, I cannot see what is the difference between that working code and that below. Everything sets up fine, I think. It gets all the way to .get(), but none of the processes ever finish.

The problem is that python3 blocks indefinitely in waiter.acquire(), which you can tell by interrupting it and reading the backtrace.

$ python3 ./try415.py
^CTraceback (most recent call last):File "./try415.py", line 43, in <module>ps = [ res.get() for res in proclist ]File "./try415.py", line 43, in <listcomp>ps = [ res.get() for res in proclist ]File "/usr/lib64/python3.6/multiprocessing/pool.py", line 638, in getself.wait(timeout)File "/usr/lib64/python3.6/multiprocessing/pool.py", line 635, in waitself._event.wait(timeout)File "/usr/lib64/python3.6/threading.py", line 551, in waitsignaled = self._cond.wait(timeout)File "/usr/lib64/python3.6/threading.py", line 295, in waitwaiter.acquire()
KeyboardInterrupt

Here's the code

from multiprocessing import Pool
from scipy import optimize
import numpy as npdef func(t, a, b, c):return 0.5*a*t**2 + b*t + cdef funcwrap(t, params):return func(t, *params)def fitWithErr(procid, yFitValues, simga, func, p0, args, bounds):np.random.seed() # force new seedrandomDelta = np.random.normal(0., sigma, len(yFitValues))randomdataY = yFitValues + randomDeltaerrfunc = lambda p, x, y: func(p, x) -yoptResult = optimize.least_squares(errfunc, p0, args=args, bounds=bounds)return optResult.xdef fit_bootstrap(function, datax, datay, p0, bounds, aprioriUnc):errfunc = lambda p, x, y: function(x,p) - yoptResult = optimize.least_squares(errfunc, x0=p0, args=(datax, datay), bounds=bounds)pfit = optResult.xresiduals = optResult.funfity = function(datax, pfit)numParallelProcesses = 2**2 # should be equal to number of ALUsnumTrials = 2**2 # this many random data sets are generated and fittedtrialParameterList = list()for i in range(0,numTrials):trialParameterList.append( [i, fity, aprioriUnc, function, p0, (datax, datay), bounds] )with Pool(processes=numParallelProcesses) as pool:proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]ps = [ res.get() for res in proclist ]ps = np.array(ps)mean_pfit = np.mean(ps,0)return mean_pfitif __name__ == '__main__':x = np.linspace(0,3,2000)p0 = [-9.81, 1., 0.]y = funcwrap(x, p0)bounds = [ (-20,-1., -1E-6),(20,3,1E-6) ]fit_bootstrap(funcwrap, x, y, p0, bounds=bounds, aprioriUnc=0.1)

Sorry for giving out the wrong answer. It's so irresponsible for not verify it. Here is the answer from me.

with Pool(processes=numParallelProcesses) as pool:

This line is wrong as with will call exit function not close. Here is exit function body:

    def __exit__(self, exc_type, exc_val, exc_tb):self.terminate()

All of the process will be terminated and never excuted. Code:

ps = [ res.get() for res in proclist ]

there is no timeout parameter. Here is the get function body:

def get(self, timeout=None):self.wait(timeout)if not self.ready():raise TimeoutErrorif self._success:return self._valueelse:raise self._value

It will always wait if no timeout. That's why it hang.

You need to change

with Pool(processes=numParallelProcesses) as pool:proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]

to:

pool=Pool(processes=numParallelProcesses)
proclist = [ pool.apply_async(fitWithErr, args) for args in trialParameterList ]
pool.close()