Python multiprocessing - Passing a list of dicts to a pool

2024/10/15 6:14:55

This question may be a duplicate. However, I read lot of stuff around on this topic, and I didn't find one that matches my case - or at least, I didn't understood it.

Sorry for the inconvenance.

What I'm trying to do is fairly common, passing a list of kwargs to pool.starmap(), to achieve multiprocessing.

Here's my reduced case:

def compute(firstArg, **kwargs): # A function that does things# Fancy computing stuff...# Saving to disk...return Trueif __name__ ==  '__main__':args = [{'firstArg': "some data",'otherArg': "some other data"'andAnotherArg': x*2} for x in range(42)]pool = Pool(4)pool.starmap(compute, args)pool.close()pool.terminate()

I supposed starmap() will unpack the dict and pass it to compute() as keyword args, but looking at the source code (see also l.46), it sends only keys (or values ?).

So it raises :

TypeError: compute() takes 1 positional argument but 3 were given

It must be a clear, straight forward way to do this... Any help would be appreciated.

Here's a quite similar question : Python Multiprocessing - How to pass kwargs to function?

Answer

You could use a tiny proxy function:

def proxy(Dict):return compute(**Dict)pool.map(proxy, args)

Or, since you don't need the proxy function polluting the namespace:

pool.map(lambda Dict: compute(**Dict), args)
https://en.xdnf.cn/q/69318.html

Related Q&A

Failed to write to file but generates no Error

Im trying to write to a file but its not working. Ive gone through step-by-step with the debugger (it goes to the write command but when I open the file its empty).My question is either: "How do I…

train spacy for text classification

After reading the docs and doing the tutorial I figured Id make a small demo. Turns out my model does not want to train. Heres the codeimport spacy import random import jsonTRAINING_DATA = [["My l…

Python threading vs. multiprocessing in Linux

Based on this question I assumed that creating new process should be almost as fast as creating new thread in Linux. However, little test showed very different result. Heres my code: from multiprocessi…

How to create a visualization for events along a timeline?

Im building a visualization with Python. There Id like to visualize fuel stops and the fuel costs of my car. Furthermore, car washes and their costs should be visualized as well as repairs. The fuel c…

Multiplying Numpy 3D arrays by 1D arrays

I am trying to multiply a 3D array by a 1D array, such that each 2D array along the 3rd (depth: d) dimension is calculated like:1D_array[d]*2D_arrayAnd I end up with an array that looks like, say:[[ [1…

Django Performing System Checks is running very slow

Out of nowhere Im running into an issue with my Django application where it runs the "Performing System Checks" command very slow. If I start the server with python manage.py runserverIt take…

str.translate vs str.replace - When to use which one?

When and why to use the former instead of the latter and vice versa?It is not entirely clear why some use the former and why some use the latter.

python BeautifulSoup searching a tag

My first post here, Im trying to find all tags in this specific html and i cant get them out, this is the code:from bs4 import BeautifulSoup from urllib import urlopenurl = "http://www.jutarnji.h…

How to remove extra whitespace from image in opencv? [duplicate]

This question already has answers here:How to remove whitespace from an image in OpenCV?(3 answers)Closed 3 years ago.I have the following image which is a receipt image and a lot of white space aroun…

Is there a way in numpy to test whether a matrix is Unitary

I was wondering if there is any function in numpy to determine whether a matrix is Unitary?This is the function I wrote but it is not working. I would be thankful if you guys can find an error in my f…