Supposed automatically threaded scipy and numpy functions arent making use of multiple cores

2024/10/14 6:18:28

I am running Mac OS X 10.6.8 and am using the Enthought Python Distribution. I want for numpy functions to take advantage of both my cores. I am having a problem similar to that of this post: multithreaded blas in python/numpy but after following through the steps of that poster, I still have the same problem. Here is my numpy.show_config():

lapack_opt_info:libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']define_macros = [('SCIPY_MKL_H', None)]include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
blas_opt_info:libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']define_macros = [('SCIPY_MKL_H', None)]include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
lapack_mkl_info:libraries = ['mkl_lapack95_lp64', 'mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']define_macros = [('SCIPY_MKL_H', None)]include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
blas_mkl_info:libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']define_macros = [('SCIPY_MKL_H', None)]include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']
mkl_info:libraries = ['mkl_intel_lp64', 'mkl_intel_thread', 'mkl_core', 'mkl_mc', 'mkl_mc3', 'pthread']library_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/lib']define_macros = [('SCIPY_MKL_H', None)]include_dirs = ['/Library/Frameworks/EPD64.framework/Versions/1.4.2/include']

As in the original post's comments, I deleted the line that set the variable MKL_NUM_THREADS=1. But even then the numpy and scipy functions that should take advantage of multi-threading are only using one of my cores at a time. Is there something else I should change?

Edit: To clarify, I am trying to get one single calculation such as numpy.dot() to use multi-threading on its own as per the MKL implementation, I am not trying to take advantage of the fact that numpy calculations release control of the GIL, hence making multi-threading with other functions easier.

Here is a small script that should make use of multi-threading but does not on my machine:

import numpy as npa = np.random.randn(1000, 10000)
b = np.random.randn(10000, 1000)np.dot(a, b) #this line should be multi-threaded
Answer

This article seems to imply that numpy intelligently makes certain operations parallel, depending on predicted speedup of the operation:

  • "If your numpy/scipy is compiled using one of these, then dot() will be computed in parallel (if this is faster) without you doing anything. "

Perhaps your small(-ish) test case won't show significant speedup according to numpy's heuristic for determining when to parallelize a particular dot() call? Maybe try a ridiculously large operation and see if both cores are utilized?

As a side note, does your processor/machine configuration actually support BLAS?

https://en.xdnf.cn/q/69443.html

Related Q&A

Golang net.Listen binds to port thats already in use

Port 8888 is already bound on my (OS X 10.13.5) system, by a process running inside a docker container:$ netstat -an | grep 8888 tcp6 0 0 ::1.8888 *.* LISTE…

Aiohttp, Asyncio: RuntimeError: Event loop is closed

I have two scripts, scraper.py and db_control.py. In scraper.py I have something like this: ... def scrape(category, field, pages, search, use_proxy, proxy_file):...loop = asyncio.get_event_loop()to_do…

Python for ios interpreter [duplicate]

This question already has answers here:Closed 11 years ago.Possible Duplicate:Python or Ruby Interpreter on iOS I just discovered this apps pypad and python for ios They have like an interpreter an ed…

Detect when multiprocessing queue is empty and closed

Lets say I have two processes: a reader and a writer. How does the writer detect when the reader has finished writing values?The multiprocessing module has a queue with a close method that seems custo…

imshow and histogram2d: cant get them to work

Im learning Python and this is my first question here. Ive read other topics related to the usage of imshow but didnt find anything useful. Sorry for my bad English.I have plotted a set of points here,…

3D Waterfall Plot with Colored Heights

Im trying to visualise a dataset in 3D which consists of a time series (along y) of x-z data, using Python and Matplotlib.Id like to create a plot like the one below (which was made in Python: http://a…

python xlsxwriter: Keep header in excel when adding a table

I have a panda dataframe that I write to a xslx file, and would like to add a table over that data. I would also like to keep the headers that I have already written, instead of adding them again. Is t…

Unexpected tokens in !DOCTYPE html in PyCharm Community Edition

I am new in using PyCharm but I am loving it gradually. I am getting a red underline on <!DOCTYPE html> and the error is "Unexpected Token".Why PyCharm shows it? I cant understand.

Indexing a numpy array with a numpy array of indexes [duplicate]

This question already has answers here:Indexing a numpy array with a list of tuples(2 answers)Index multidimensional array with index array(1 answer)Closed 5 years ago.I have a 3D numpy array data and …

Input redirection with python

I have the following program to test input redirection in Python.a = int(raw_input("Enter a number: ")) b = raw_input("Enter a string: ") print "number entered = ", a prin…