Python multiprocessing with M1 Mac

2024/7/27 14:54:02

I have a mac (Mac Os 11.1, Python Ver 3.8.2) and need to work in multiprocessing, but the procedures doesn’t work.

import multiprocessingdef func(index: int):print(index)manager = multiprocessing.Manager()
processes = []for i in range(-1, 10):p = multiprocessing.Process(target=func,args=(i,))processes.append(p)p.start()for process in processes:process.join()

However, on my Intel-based Mac, it works fine.

What I expect is

-1
0
1
2
3
4
5
6
7
8
9

But instead, I got an error:

RuntimeError: An attempt has been made to start a new process before thecurrent process has finished its bootstrapping phase.This probably means that you are not using fork to start yourchild processes and you have forgotten to use the proper idiomin the main module:if __name__ == '__main__':freeze_support()...The "freeze_support()" line can be omitted if the programis not going to be frozen to produce an executable.
Traceback (most recent call last):File "/Users/lance/Documents/Carleton Master Projects/Carleton-Master-Thesis/experiment.py", line 7, in <module>manager = multiprocessing.Manager()File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/context.py", line 57, in Managerm.start()File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/managers.py", line 583, in startself._address = reader.recv()File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 250, in recvbuf = self._recv_bytes()File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 414, in _recv_bytesbuf = self._recv(4)File "/Library/Developer/CommandLineTools/Library/Frameworks/Python3.framework/Versions/3.8/lib/python3.8/multiprocessing/connection.py", line 383, in _recvraise EOFError
EOFError

Is there any similar way (also keep it easy) to parallelize in M1-based Mac?

Answer

I'm not sure why this works on an Intel machine, but the problem is all the MP-related code should be inside if __name__ == '__main__':

import multiprocessingdef func(index: int):print(index)if __name__ == '__main__':manager = multiprocessing.Manager()processes = []for i in range(-1, 10):p = multiprocessing.Process(target=func, args=(i,))processes.append(p)p.start()for process in processes:process.join()

Since MP actually starts a new process, the new process needs to import the func function from this module to execute its task. If you don't protect the code that starts new processes, it will be executed during this import.

https://en.xdnf.cn/q/72885.html

Related Q&A

What is faster in Python, while or for xrange

We can do numeric iteration like:for i in xrange(10):print i,and in C-style:i = 0 while i < 10:print i,i = i + 1Yes, I know, the first one is less error-prone, more pythonic but is it fast enough as…

Concatenate Numpy arrays with least memory

Not I have 50GB dataset saved as h5py, which is a dictionary inside. The dictionary contains keys from 0 to n, and the values are numpy ndarray(3 dimension) which have the same shape. For example:dicti…

How to generate random programs from BNF

I know my question sounds a little vague, but I could not find any tutorials online. I am not asking for an answer, but for more of an explanation. An example of the BNF:<prog> ::= “int main() {…

Pandas: merge multiple dataframes and control column names?

I would like to merge nine Pandas dataframes together into a single dataframe, doing a join on two columns, controlling the column names. Is this possible?I have nine datasets. All of them have the fo…

Two different plots from same loop in matplotlib?

I would specifically like to create two different plots using one single loop. One plot should have four straight lines from x-y, and another plot should have four angled lines from x-y2. My code only …

Matplotlib text alignment

Is there a way to get the result shown in the third axes with just a single ax.text() command? Using expandtabs almost get me there, but the text never aligns properly. Using two plotting commands doe…

Pandas cannot load data, csv encoding mystery

I am trying to load a dataset into pandas and cannot get seem to get past step 1. I am new so please forgive if this is obvious, I have searched previous topics and not found an answer. The data is mos…

How to read in an edge list to make a scipy sparse matrix

I have a large file where each line has a pair of 8 character strings. Something like:ab1234gh iu9240ghon each line.This file really represents a graph and each string is a node id. I would like to r…

How can I find the best fuzzy string match?

Pythons new regex module supports fuzzy string matching. Sing praises aloud (now). Per the docs:The ENHANCEMATCH flag makes fuzzy matching attempt to improve the fitof the next match that it finds.The …

how to write a unicode csv in Python 2.7

I want to write data to files where a row from a CSV should look like this list (directly from the Python console):row = [\xef\xbb\xbft_11651497, http://kozbeszerzes.ceu.hu/entity/t/11651497.xml, "…