Is there an easy way to tell how much time is spent waiting for the Python GIL?

2024/10/11 17:28:12

I have a long-running Python service and I'd like to know how much cumulative wall clock time has been spent by any runnable threads (i.e., threads that weren't blocked for some other reason) waiting for the GIL. Is there an easy way to do this? E.g., perhaps I could periodically dump some counter to its log file.

My underlying motivation is to rule out the GIL as a source of mystery response latency from these long-running processes. There is no particular reason to suspect the GIL (other than that it would fit the symptoms), but other forms of logging haven't turned up anything yet, so, if it is easy, it would be nice to have this information.

Answer

I don't think there's an easy way. There's probably an awkward way, involving rebuilding Python to traverse the PyThreadState list and count the threads each time the lock is acquired, but I doubt it's worth the effort!

I know this is a speculative question but if you are even moderately concerned about there being delays caused by threading it may be prudent to move to a multiprocessing model instead of a multithreading model. Since processes are both safer and more scalable in Python they are almost always the best choice if practical.

https://en.xdnf.cn/q/69748.html

Related Q&A

Inverse filtering using Python

Given an impulse response h and output y (both one-dimensional arrays), Im trying to find a way to compute the inverse filter x such that h * x = y, where * denotes the convolution product.For example,…

Quadruple Precision Eigenvalues, Eigenvectors and Matrix Logarithms

I am attempting to diagonalize matrices in quadruple precision, and to take their logarithms. Is there a language in which I can accomplish this using built-in functions?Note, the languages/packages i…

How to use pyinstaller with pipenv / pyenv

I am trying to ship an executable from my python script which lives inside a virtual environment using pipenv which again relies on pyenv for python versioning. For that, I want to us pyinstaller. Wha…

Sending DHCP Discover using python scapy

I am new to python and learning some network programming, I wish to send an DHCP Packet through my tap interface to my DHCP server and expecting some response from it. I tried with several packet build…

cnf argument for tkinter widgets

So, Im digging through the code here and in every class (almost) I see an argument cnf={} to the constructor, but unless Ive missed it, it is not explicitly stated what cnf is / expected to contain. Ca…

python exceptions.UnicodeDecodeError: ascii codec cant decode byte 0xa7 in

I am using scrapy with python and I have this code in a python item piplinedef process_item(self, item, spider):import pdb; pdb.set_trace()ID = str(uuid.uuid5(uuid.NAMESPACE_DNS, item[link]))I got this…

Trace Bug which happends only sometimes in CI

I have a strange bug in python code which only happens sometimes in CI.We cant reproduce it.Where is the test code:response=self.admin_client.post(url, post) self.assertEqual(200, response.status_code,…

Limit neural network output to subset of trained classes

Is it possible to pass a vector to a trained neural network so it only chooses from a subset of the classes it was trained to recognize. For example, I have a network trained to recognize numbers and l…

Unable to fully remove border of PyQt QGraphicsView

I have tried calling self.setStyleSheet("background: transparent; border: transparent;") on a QGraphicsView, but it still leaves a 1 pixel border on the top edge. I have also tried replacing …

SqlAlchemy non persistent column

I am trying to define a model using sqlalchemy such that one of the columns i only need in memory for processing, i do not have a corresponding database column for that. I need it such that when i save…