the design and goal of monarch

[eric-haibin-lin]

hi, this is Haibin from verl team. monarch looks very interesting. What is the goal/non-goal of monarch, compared with ray, and is there a design principle for devs in the community to learn more?

[mariusae]

Monarch is our attempt at delivering a “single controller” style cluster training model for PyTorch: we want to make orchestrating training jobs spanning large clusters of GPUs as easy as writing a local PyTorch program – as if your machine had thousands of GPUs attached to it.

As such, we are laser focused on solving the performance and scalability challenges that come along with this goal. Thus, a design principle of Monarch is that operations spanning large GPU clusters should be cheap. Our model for this is that Monarch’s core computing widgets – processes, actors, tensors – are organized into large collections called meshes. You can almost think of this as “distributed SIMD”, allowing the controller to efficiently issue commands to large meshes of processes, actors, or tensors.

So, in this way you can think of Monarch’s model almost as a “vectorized Ray”, but also with other important differences, for example:

• The error models are different. Monarch’s error handling is aligned with meshes, and not individual actors. Error recovery follows a similar path.
• Monarch does not provide an object store. This reduces flexibility somewhat, but also simplifies the model and eases scaling.
• Monarch has native RDMA support, wherein actors can exchange data through RDMA buffers that support one-sided transfers.
• Monarch is more explicitly layered, to provide control to the user. For example, Monarch gives users control over launching processes, and actors within the processes; Monarch’s tensor engine is built on top of actors in turn.
• A native PyTorch tensor engine provides seamless integration between batch-oriented tensor compute (e.g., training) and asynchronous actors. For example, Monarch enables seamless transfer and resharding of parameters between trainers (trained with Monarch’s tensor engine) and inference servers (e.g., using vLLM), all while utilizing RDMA for backend transfers.

An important non-goal is to support workloads other than ML training. We are laser focused on this use case, and it deeply informs Monarch’s design decisions and trade-offs.

Note that Monarch is still in a great deal of flux! We open sourced it early so that we can begin to take dependencies on the outside. We are working through a number of design changes which will be evident soon. Our plan after that is to much more thoroughly document the system, and open it up for wider use. Thus, Monarch’s APIs (and implementation) will be highly unstable for the next little bit, and we highly discourage taking dependencies on the project until we announce a beta version.

[suo]

I will note there are a lot of similarities with the hybridflow design and how we think about meshes—the goal of meshes is to enable very efficient message dispatch to SPMD (i.e. multi-process) groups of compute, while retaining single controller properties for control flow at the top level.

[terrykong]

Hello monarch team! Congrats on the release! How does the monarch team view things beyond the torch environment like environments / tool calling / verifiers, especially in cases where the environment may be a very different isolated environment? Is that within scope of monarch?

[mariusae]

How does the monarch team view things beyond the torch environment like environments / tool calling / verifiers, especially in cases where the environment may be a very different isolated environment? Is that within scope of monarch?

Yes, this is in scope. Monarch provides a general purpose (vectorized) actor abstraction that is intended to run arbitrary code such as RL environments and tools.