-
-
Notifications
You must be signed in to change notification settings - Fork 186
multiprocessing fork() deprecation warning on Python 3.12+ #480
Copy link
Copy link
Open
Description
Description
On Python 3.12+, running tasks with the multiprocessing runner (MRunner) produces a DeprecationWarning:
/usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=197896) is multi-threaded, use of fork() may lead to deadlocks in the child.
self.pid = os.fork()
This happens because the multiprocessing module defaults to fork on Linux, and if any threads are running, fork() can lead to deadlocks in the child process.
Background
Python 3.12 added this warning to flag unsafe usage of fork() in multi-threaded processes. Python 3.14 is expected to change the default start method from fork to forkserver on Linux.
Possible approaches
spawn: safest, but requires all task data to be picklable (related tocloudpickleoptional dependency)forkserver: middle ground, avoids the fork-in-threads issue but has its own quirks- Make it configurable: let users choose the start method via
DOIT_CONFIG
Considerations
- Changing the default from
forkcould break existing users'dodo.pyfiles that rely on fork behavior (shared state, file handles, etc.) - This is a backward-incompatible change that needs careful evaluation
- Related:
cloudpickleis already an optional dependency for pickling support withspawn
References
- https://docs.python.org/3.12/library/multiprocessing.html#contexts-and-start-methods
- Related file:
doit/runner.py(MRunnerclass)
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels