ENH: Use multi-phase initialisation (PEP 489) for extension modules

[AA-Turner]

Proposed new feature or change:

This is intended to be a tracking issue for a series of PRs I will shortly submit.

Multi-phase initialisation was introduced in PEP 489 for Python 3.5. It replaces the previous 'single-phase' mechanism by splitting the module creation process into creation and execution phases. Quoting from the documentation:

Extension modules created this way behave more like Python modules: the initialization is split between the creation phase, when the module object is created, and the execution phase, when it is populated. The distinction is similar to the __new__() and __init__() methods of classes.

This is also the stable mechanism to declare compatibility with both free-threading and subinterpreters (though note that this PR does not attempt to add support for subinterpreters).

A common point of confusion/conflation is multi-phase initialisation and module isolation. As noted on the Python Discourse forum, "Implementing extension module isolation is a very good thing, but if a module uses static types or (C) global variables then the effort to isolate becomes non-trivial.". This issue only seeks to implement multi-phase initialisation, generally following the guide that Eric set out in that thread:

1. move the content of the module’s init function to a corresponding “module exec” function (dropping the call to create the module object)
2. set the module def’s m_slots field to an array with:

• a Py_mod_exec slot, set to the new module exec function
• a Py_mod_multiple_interpreters slot, set to Py_MOD_MULTIPLE_INTERPRETERS_NOT_SUPPORTED
• to try free-threading, a Py_mod_gil slot, set to Py_MOD_GIL_NOT_USED

3. set def.m_size to 0 (if negative)
4. update the module init function to only return PyModuleDef_Init(def); for the corresponding module def

cc @ngoldbaum

xref:

• ENH: Support free-threaded python build (tracking issue) #26157 (comment)
• BUG: Cannot import numpy from different interpreters in the same process #28271 (comment)
• ENH: Please support subinterpreters #24755
• segfault when importing numpy after reinitializing Python #11925 (comment)
• Deadlock happening when using numpy/scipy with python subinterpreters (Trac #2213) #665

A

PRs:

• GH-29021: Convert multiarray to multi-phase init (PEP 489) #29022
• GH-29021: Convert umath to multi-phase init (PEP 489) #29023
• GH-29021: Convert the ufunc tutorial to multi-phase init (PEP 489) #29024
• GH-29021: Convert _simd.c to multi-phase init (PEP 489) #29025
• GH-29021: Convert dummymodule to multi-phase init (PEP 489) #29026
• GH-29021: Convert limited API examples to multi-phase init (PEP 489) #29027
• GH-29021: Convert pocketfft_umath to multi-phase init (PEP 489) #29028
• GH-29021: Convert lapack_lite to multi-phase init (PEP 489) #29029
• GH-29021: Convert umath_linalg to multi-phase init (PEP 489) #29030
• GH-29021: Convert extbuild to multi-phase init (PEP 489) #29031
• TODO: simd_dispatch
• TODO: f2py