Python code calls C library that create OS threads, which eventually call Python callbacks

2024/4/15 1:08:06

If the one and only Python interpreter is in the middle of executing a bytecode when the OS dispatches another thread, which calls a Python callback - what happens? Am I right to be concerned about the reliability of this design?


In general situations, the C library needs to call PyEval_InitThreads() to gain GIL before spawning any thread that invokes python callbacks. And the callbacks need to be surrounded with PyGILState_Ensure() and PyGILState_Release() to ensure safe execution.

However, if the C library is running within the context of, say, a python C extension, then there are simple cases where it is safe to omit GIL manipulation at all.

Think of this calling sequence: 1) python code calls C function foo(), 2) foo() spawns one and only one thread that runs another C function bar(), which calls back to python code, and 3) foo() always joins or cancels the thread running bar() before returns.

In such a case, it is safe to omit GIL manipulation. Because foo() owns the GIL (i.e. implicitly borrowed from the python code that calls it) during its lifecycle, and the execution of python callbacks within the lifecycle of foo() is serialized (i.e. only one callback thread and python code does not incorporate threading).

Related Q&A

Django MTMField: limit_choices_to = other_ForeignKeyField_on_same_model?

Ive got a couple django models that look like this:from django.contrib.sites.models import Siteclass Photo(models.Model):title = models.CharField(max_length=100)site = models.ForeignKey(Site)file = mod…

Logging django.request to file instead of console

I am trying to configure my django to properly use the python logging facility but Ive stumbled upon a rather strange problem:Even after reading the docs, I simply cant find out how to redi…

Coinbase APIerror(id = ) in python

I want to transfer money between my coinbase accounts. Im storing all of my accounts IDs from client.get_accounts()[data][id] and transferring with the code, tx = client.transfer_money(2bbf394c-193b-5b…

cxfreeze missing distutils module inside virtualenv

When running a cxfreeze binary from a python3.2 project I am getting the following runtime error:/project/dist/project/distutils/ UserWarning: The virtualenv distutils package at %s appe…

Preventing a multiplication expression evaluating in Sympy

I am generating an expression with two fractions, and want to pretty print as a whole expression with LaTeX, to then put on a worksheet.E.g. in the form:(5/7) * (3/4). However, when I do the following:…

Calling PARI/GP from Python

I would like to call PARI/GP from Python only to calculate the function nextprime(n) for different ns that I define. Unfortunately I cant get pari-python to install so I thought I would just call it us…

Can anyone explain why this sorting wont work?

For example if I have a list like this:List1 =[7,6,9] List1 = List1.sort()

Using argparse for mandatory argument without prefix

Im trying to use argparse module within my python application. My application should be run with single mandatory argument without any prefixes. I could not come up with a way to do it.

prevent unexpected stdin reads and lock in subprocess

A simple case Im trying to solve for all situations. I am running a subprocess for performing a certain task, and I dont expect it to ask for stdin, but in rare cases that I might not even expect, it m…

Regex split string by last occurrence of pattern

I am using regex to split a string <book name> by <author name> into book and author\bby\b, text, 0, re.I)But problem arises when the book name contains the word "by&q…