How to identify the CPU core ID of a process on Python multiprocessing?

2024/10/13 5:21:30

I am testing Python's multiprocessing module on a cluster with SLURM. I want to make absolutely sure that each of my tasks are actually running on separate cpu cores as I intend. Due to the many possibilities of configuring SLURM, this is not at all obvious.

Therefore, I was wondering if there is a way to get core specific information from the running Python task. I need my Python script to get information about the core it's running on which allows to distinguish between the various cores.

Consider the following script. There are 10 tasks launched. Is there a way for each of them to print core specific information about the core they have been assigned to?

import multiprocessingdef hello():print "Hello World"pool = multiprocessing.Pool()
jobs = []
for j in range(10):p = multiprocessing.Process(target = hello)jobs.append(p)p.start()

The modules I found here and here give CPU specific information, but these are about available CPUs, not core information at runtime.

Answer

On Linux, FreeBSD, SunOS you can use the psutil-module for that:

psutil.Process().cpu_num()

Note, that as long as you don't set the cpu-affinity as well, your OS will arbitrarily migrate the execution to different cores. Processes are not assigned to specific cores by default and it's usually best to let your OS decide on which core to run something.

https://en.xdnf.cn/q/69566.html

Related Q&A

Finding highest values in each row in a data frame for python

Id like to find the highest values in each row and return the column header for the value in python. For example, Id like to find the top two in each row:df = A B C D 5 9 8 2 4 …

Using pytest_addoptions in a non-root conftest.py

I have a project that has the following structure: Project/ | +-- src/ | | | +-- proj/ | | | +-- __init__.py | +-- code.py | +-- tests/ | | | +-- __init_…

How to count distinct values in a combination of columns while grouping by in pandas?

I have a pandas data frame. I want to group it by using one combination of columns and count distinct values of another combination of columns.For example I have the following data frame:a b c …

Set Environmental Variables in Python with Popen

I want to set an environmental variable in linux terminal through a python script. I seem to be able to set environmental variables when using os.environ[BLASTDB] = /path/to/directory .However I was in…

Python - pandas - Append Series into Blank DataFrame

Say I have two pandas Series in python:import pandas as pd h = pd.Series([g,4,2,1,1]) g = pd.Series([1,6,5,4,"abc"])I can create a DataFrame with just h and then append g to it:df = pd.DataFr…

How to retrieve values from a function run in parallel processes?

The Multiprocessing module is quite confusing for python beginners specially for those who have just migrated from MATLAB and are made lazy with its parallel computing toolbox. I have the following fun…

SignalR Alternative for Python

What would be an alternative for SignalR in Python world?To be precise, I am using tornado with python 2.7.6 on Windows 8; and I found sockjs-tornado (Python noob; sorry for any inconveniences). But s…

Python Variable Scope and Classes

In Python, if I define a variable:my_var = (1,2,3)and try to access it in __init__ function of a class:class MyClass:def __init__(self):print my_varI can access it and print my_var without stating (glo…

How to check if valid excel file in python xlrd library

Is there any way with xlrd library to check if the file you use is a valid excel file? I know theres other libraries to check headers of files and I could use file extension check. But for the sake of…

ValueError: Invalid file path or buffer object type: class tkinter.StringVar

Here is a simplified version of some code that I have. In the first frame, the user selects a csv file using tk.filedialog and it is meant to be plotted on the same frame on the canvas. There is also a…