Why cant I use operator.itemgetter in a multiprocessing.Pool?

2024/9/28 17:25:23

The following program:

import multiprocessing,operator
f = operator.itemgetter(0)
# def f(*a): return operator.itemgetter(0)(*a)
if __name__ == '__main__':multiprocessing.Pool(1).map(f, ["ab"])

fails with the following error:

Process PoolWorker-1:
Traceback (most recent call last):File "/usr/lib/python3.2/multiprocessing/process.py", line 267, in _bootstrapself.run()File "/usr/lib/python3.2/multiprocessing/process.py", line 116, in runself._target(*self._args, **self._kwargs)File "/usr/lib/python3.2/multiprocessing/pool.py", line 102, in workertask = get()File "/usr/lib/python3.2/multiprocessing/queues.py", line 382, in getreturn recv()
TypeError: itemgetter expected 1 arguments, got 0

Why do I get the error (on cPython 2.7 and 3.2 on Linux x64), and why does it vanish if I uncomment the third line?

Answer

The problem here is that the multiprocessing module passes objects by copy into the other processes (obviously), and itemgetter objects are not copyable using any of the obvious means:

In [10]: a = operator.itemgetter(0)
Out[10]: copy.copy(a)
TypeError: itemgetter expected 1 arguments, got 0In [10]: a = operator.itemgetter(0)
Out[10]: copy.deepcopy(a)
TypeError: itemgetter expected 1 arguments, got 0In [10]: a = operator.itemgetter(0)
Out[10]: pickle.dumps(a)
TypeError: can't pickle itemgetter objects# etc.

The problem isn't even attempting to call f inside the other processes; it's trying to copy it in the first place. (If you look at the stack traces, which I omitted above, you'll see a lot more information on why this fails.)

Of course usually this doesn't matter, because it's nearly as easy and efficient to construct a new itemgetter on the fly as to copy one. And this is what your alternative "f" function is doing. (Copying a function that creates an itemgetter on the fly doesn't require copying an itemgetter, of course.)

You could turn "f" into a lambda. Or write a trivial function (named or lambda) that does the same thing without using itemgetter. Or write an itemgetter replacement that is copyable (which obviously wouldn't be all that hard). But you can't directly use itemgetter objects as-is the way you want to.

https://en.xdnf.cn/q/71309.html

Related Q&A

Writing a compiler for a DSL in python

I am writing a game in python and have decided to create a DSL for the map data files. I know I could write my own parser with regex, but I am wondering if there are existing python tools which can do …

How to setup Celery to talk ssl to Azure Redis Instance

Using the great answer to "How to configure celery-redis in django project on microsoft azure?", I can configure Celery to use Azure Redis Cache using the non-ssl port, 6379, using the follo…

Cant save data from yfinance into a CSV file

I found library that allows me to get data from yahoo finance very efficiently. Its a wonderful library.The problem is, I cant save the data into a csv file.Ive tried converting the data to a Panda Da…

silhouette coefficient in python with sklearn

Im having trouble computing the silhouette coefficient in python with sklearn. Here is my code :from sklearn import datasets from sklearn.metrics import * iris = datasets.load_iris() X = pd.DataFrame(i…

Force dask to_parquet to write single file

When using dask.to_parquet(df, filename) a subfolder filename is created and several files are written to that folder, whereas pandas.to_parquet(df, filename) writes exactly one file. Can I use dasks t…

Unable to get python embedded to work with zipd library

Im trying to embed python, and provide the dll and a zip of the python libraries and not use any installed python. That is, if a user doesnt have python, I want my code to work using the provided dll/…

Convert integer to a random but deterministically repeatable choice

How do I convert an unsigned integer (representing a user ID) to a random looking but actually a deterministically repeatable choice? The choice must be selected with equal probability (irrespective o…

Using python opencv to load image from zip

I am able to successfully load an image from a zip:with zipfile.ZipFile(test.zip, r) as zfile:data = zfile.read(test.jpg)# how to open this using imread or imdecode?The question is: how can I open thi…

DeprecationWarning: Function when moving app (removed titlebar) - PySide6

I get when I move the App this Warning: C:\Qt\Login_Test\main.py:48: DeprecationWarning: Function: globalPos() const is marked as deprecated, please check the documentation for more information.self.dr…

Architecture solution for Python Web application

Were setting up a Python REST web application. Right now, were using WSGI, but we might do some changes to that in the future (using Twisted, for example, to improve on scalability or some other featu…