Python multiprocessing returning AttributeError when following documentation code [duplicate]

2024/10/8 19:51:07

I decided to try and get into the multiprocessor module to help speed up my program. To figure it out, I tried using some of the code examples on the official python documentation on multiprocessing.

First attempt: Introduction

>>> from multiprocessing import Pool
>>>
>>> def f(x):
...     return x*x
...
>>> if __name__ == '__main__':
...     with Pool(5) as p:
...         print(p.map(f, [1, 2, 3]))
...
Process SpawnPoolWorker-3:
Process SpawnPoolWorker-2:
Traceback (most recent call last):File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrapself.run()File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in runself._target(*self._args, **self._kwargs)File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in workertask = get()File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in getreturn _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Traceback (most recent call last):File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrapself.run()File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in runself._target(*self._args, **self._kwargs)File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in workertask = get()File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in getreturn _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>
Process SpawnPoolWorker-4:
Traceback (most recent call last):File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 258, in _bootstrapself.run()File "C:\Program Files\Python36\lib\multiprocessing\process.py", line 93, in runself._target(*self._args, **self._kwargs)File "C:\Program Files\Python36\lib\multiprocessing\pool.py", line 108, in workertask = get()File "C:\Program Files\Python36\lib\multiprocessing\queues.py", line 337, in getreturn _ForkingPickler.loads(res)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

Here I assume that the pool function is broken; maybe there is a typo somewhere in the lastest version. So I try some of the more specific code.

Second attempt: Process class code block 2

>>> from multiprocessing import Process
>>> import os
>>>
>>> def info(title):
...     print(title)
...     print('module name:', __name__)
...     print('parent process:', os.getppid())
...     print('process id:', os.getpid())
...
>>> def f(name):
...     info('function f')
...     print('hello', name)
...
>>> if __name__ == '__main__':
...     info('main line')
...     p = Process(target=f, args=('bob',))
...     p.start()
...     p.join()
...
main line
module name: __main__
parent process: 43824
process id: 54888
Traceback (most recent call last):File "<string>", line 1, in <module>File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_mainexitcode = _main(fd)File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _mainself = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

At this point I know the underlying error is with the Process function of multiprocessing. However, I think that the extended code might have broken something, so I try the simple code.

Third Attempt Process class code block 1

>>> from multiprocessing import Process
>>>
>>> def f(name):
...     print('hello', name)
...
>>> if __name__ == '__main__':
...     p = Process(target=f, args=('bob',))
...     p.start()
...     p.join()
...
Traceback (most recent call last):File "<string>", line 1, in <module>File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_mainexitcode = _main(fd)File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _mainself = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

At this point I was desperate. I think that maybe the argument was messing with the Process class.

Final attempt: self-generated code

>>> from multiprocessing import Process
>>>
>>> def f():
...     print('hello')
...
>>> if __name__ == '__main__':
...     p = Process(target=f)
...     p.start()
...     p.join()
...
Traceback (most recent call last):File "<string>", line 1, in <module>File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 105, in spawn_mainexitcode = _main(fd)File "C:\Program Files\Python36\lib\multiprocessing\spawn.py", line 115, in _mainself = reduction.pickle.load(from_parent)
AttributeError: Can't get attribute 'f' on <module '__main__' (built-in)>

Now I am totally confused because I do not know why the error is occuring. Could someone help me figure out why I am getting this error every time?

Answer

You're in interactive mode. That basically doesn't work with multiprocessing, because the workers have to import __main__ and get something that mostly resembles the main process's __main__. This is one of the many ways in which the multiprocessing API is horribly confusing.

Put your code in a script and run the script.

https://en.xdnf.cn/q/70099.html

Related Q&A

Python - How can I find if an item exists in multidimensional array?

Ive tried a few approaches, none of which seem to work for me. board = [[0,0,0,0],[0,0,0,0]]if not 0 in board:# the board is "full"I then tried:if not 0 in board[0] or not 0 in board[1]:# the…

Convert Geo json with nested lists to pandas dataframe

Ive a massive geo json in this form:{features: [{properties: {MARKET: Albany,geometry: {coordinates: [[[-74.264948, 42.419877, 0],[-74.262041, 42.425856, 0],[-74.261175, 42.427631, 0],[-74.260384, 42.4…

Pymongo - ValueError: NaTType does not support utcoffset when using insert_many

I am trying to incrementally copy documents from one database to another. Some fields contain date time values in the following format:2016-09-22 00:00:00while others are in this format:2016-09-27 09:0…

python numpy argmax to max in multidimensional array

I have the following code:import numpy as np sample = np.random.random((10,10,3)) argmax_indices = np.argmax(sample, axis=2)i.e. I take the argmax along axis=2 and it gives me a (10,10) matrix. Now, I …

Can Keras model.predict return a dictionary?

The documentation https://keras.io/models/model/#predict says that model.predict returns Numpy array(s) of predictions. In the Keras API, is there is a way to distinguishing which of these arrays are…

Flask OIDC: oauth2client.client.FlowExchangeError

The Problem: The library flask-oidc includes the scope parameter into the authorization-code/access-token exchange request, which unsurprisingly throws the following error:oauth2client.client.FlowExcha…

Cumulative count at a group level Python

I have a pandas dataframe like this : df = pd.DataFrame([[A, 1234, 20120201],[A, 1134, 20120201],[A, 1011, 20120201],[A, 1123, 20121004],[A, 1111, 20121004],[A, 1224, 20121105],[B, 1156, 20120403],[B, …

Easiest ways to generate graphs from Python? [closed]

Closed. This question is seeking recommendations for books, tools, software libraries, and more. It does not meet Stack Overflow guidelines. It is not currently accepting answers.We don’t allow questi…

Stripping python namespace attributes from an lxml.objectify.ObjectifiedElement [duplicate]

This question already has answers here:Closed 11 years ago.Possible Duplicate:When using lxml, can the XML be rendered without namespace attributes? How can I strip the python attributes from an lxml…

matplotlib xkcd and black figure background

I am trying to make a plot using matplotlibs xkcd package while having a black background. However, xkcd seems to add a sort of white contour line around text and lines. On a white background you cant …