Where is Pythons shutdown procedure setting module globals to None documented?

2024/11/20 6:18:33

CPython has a strange behaviour where it sets modules to None during shutdown. This screws up error logging during shutdown of some multithreading code I've written.

I can't find any documentation of this behaviour. It's mentioned in passing in PEP 432:

[...] significantly reducing the number of modules that will experience the "module globals set to None" behaviour that is used to deliberate break cycles and attempt to releases more external resources cleanly.

There are SO questions about this behaviour and the C API documentation mentions shutdown behaviour for embedded interpreters.

I've also found a related thread on python-dev and a related CPython bug:

This patch does not change the behavior of moduleobjects clearing their globals dictionary as soon asthey are deallocated.

Where is this behaviour documented? Is it Python 2 specific?

Answer

The behaviour is not well documented, and is present in all versions of Python from about 1.5-ish until Python 3.4:

As part of this change, module globals are no longer forcibly set to None during interpreter shutdown in most cases, instead relying on the normal operation of the cyclic garbage collector.

The only documentation for the behaviour is the moduleobject.c source code:

/* To make the execution order of destructors for globalobjects a bit more predictable, we first zap all objectswhose name starts with a single underscore, before we clearthe entire dictionary.  We zap them by replacing them withNone, rather than deleting them from the dictionary, toavoid rehashing the dictionary (to some extent). */

Note that setting the values to None is an optimisation; the alternative would be to delete names from the mapping, which would lead to different errors (NameError exceptions rather than AttributeErrors when trying to use globals from a __del__ handler).

As you found out on the mailinglist, the behaviour predates the cyclic garbage collector; it was added in 1998, while the cyclic garbage collector was added in 2000. Since function objects always reference the module __dict__ all function objects in a module involve circular references, which is why the __dict__ needed clearing before GC came into play.

It was kept in place even when cyclic GC was added, because there might be objects with __del__ methods involved in cycles. These aren't otherwise garbage-collectable, and cleaning out the module dictionary would at least remove the module __dict__ from such cycles. Not doing that would keep all referenced globals of that module alive.

The changes made for PEP 442 now make it possible for the garbage collector to clear cyclic references with objects that provide a __del__ finalizer, removing the need to clear the module __dict__ for most cases. The code is still there but this is only triggered if the __dict__ attribute is still alive even after moving the contents of sys.modules to weak references and starting a GC collection run when the interpreter is shutting down; the module finalizer simply decrements their reference count.

https://en.xdnf.cn/q/26356.html

Related Q&A

Save a dictionary to a file (alternative to pickle) in Python?

Answered I ended up going with pickle at the end anywayOk so with some advice on another question I asked I was told to use pickle to save a dictionary to a file. The dictionary that I was trying to sa…

python equivalent of functools partial for a class / constructor

I want to create a class that behaves like collections.defaultdict, without having the usage code specify the factory. EG: instead of class Config(collections.defaultdict):passthis:Config = functools.p…

Set the font size in pycharms python console or terminal

There are terminal and python console in pycharm, which are very convenient. But I found that the font size was too small to recognize in terminal or python console. How can change the font size in the…

How to filter model results for multiple values for a many to many field in django

I have the following Model:class Group(models.Model):member = models.ManyToManyField(Player, through=GroupMember)name = models.CharField(max_length=20, unique=True)join_password = models.CharField(max_…

Why is string comparison so fast in python?

I became curious to understand the internals of how string comparison works in python when I was solving the following example algorithm problem:Given two strings, return the length of the longest comm…

What is a namespace object?

import argparseparser = argparse.ArgumentParser(description=sort given numbers) parser.add_argument(-s, nargs = +, type = int) args = parser.parse_args() print(args)On command line when I run the comma…

How to get the element-wise mean of an ndarray

Id like to calculate element-wise average of numpy ndarray.In [56]: a = np.array([10, 20, 30])In [57]: b = np.array([30, 20, 20])In [58]: c = np.array([50, 20, 40])What I want:[30, 20, 30]Is there any …

Spark 1.4 increase maxResultSize memory

I am using Spark 1.4 for my research and struggling with the memory settings. My machine has 16GB of memory so no problem there since the size of my file is only 300MB. Although, when I try to convert …

Java abstract/interface design in Python

I have a number of classes which all share the same methods, only with different implementations. In Java, it would make sense to have each of these classes implement an interface or extend an abstract…

PyCharm - no tests were found?

Ive been getting na error in PyCharm and I cant figure out why Im getting it:No tests were foundThis is what I have for my point_test.py: import unittest import sys import ossys.path.insert(0, os.path.…