Multi node example

[hemildesai]

Hi Monarch team, are there plans to add examples for running monarch on multiple nodes/hosts?

[hemildesai]

I guess there's a python binding here - #170.

But would be good to have a example showcasing how to add different remote servers to the cluster.

[hemildesai]

I was able to build the process_allocator binary, looks like that should be enough. Would be good to include the binary in the python installation.

bin/release/process_allocator --help
Runs hyperactor's process allocator

Usage: process_allocator [OPTIONS]

Options:
      --port <PORT>        The port to bind to on [::] (all network
                           interfaces on this host). Same as specifying
                           `--addr=[::]:{port}` [default: 26600]
      --addr <ADDR>        The address to bind to in the form:
                           `{transport}!{address}:{port}` (e.g.
                           `tcp!127.0.0.1:26600`). If specified, `--port`
                           argument is ignored
      --program <PROGRAM>  The path to the binary that this process
                           allocator spawns on an `allocate` request
                           [default: monarch_bootstrap]
  -h, --help               Print help

[suo]

Yes, we are working on a better example here :) @kiukchung is planning to integrate with torchx as an allocator, so that we can allocate actors on using any cluster scheduler that torchx supports (slurm, kubernetes, ray, etc.)