Skip to content

multiprocessing fork() deprecation warning on Python 3.12+ #480

@schettino72

Description

@schettino72

Description

On Python 3.12+, running tasks with the multiprocessing runner (MRunner) produces a DeprecationWarning:

/usr/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=197896) is multi-threaded, use of fork() may lead to deadlocks in the child.
    self.pid = os.fork()

This happens because the multiprocessing module defaults to fork on Linux, and if any threads are running, fork() can lead to deadlocks in the child process.

Background

Python 3.12 added this warning to flag unsafe usage of fork() in multi-threaded processes. Python 3.14 is expected to change the default start method from fork to forkserver on Linux.

Possible approaches

  • spawn: safest, but requires all task data to be picklable (related to cloudpickle optional dependency)
  • forkserver: middle ground, avoids the fork-in-threads issue but has its own quirks
  • Make it configurable: let users choose the start method via DOIT_CONFIG

Considerations

  • Changing the default from fork could break existing users' dodo.py files that rely on fork behavior (shared state, file handles, etc.)
  • This is a backward-incompatible change that needs careful evaluation
  • Related: cloudpickle is already an optional dependency for pickling support with spawn

References

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions