Prefect serverless work pool executor

[TomNicholas]

One of Matt Rocklin's blog posts mentions Prefect's "serverless work pool"

https://docs.prefect.io/v3/deploy/infrastructure-examples/serverless

My (hazy) understanding of what this is is a way to deploy a cluster in the cloud on-demand and submit work to it without having a client process controlling it at all times. This seems like it could be useful as a way to provision a cluster on which to run an in-memory rechunk without having to worry about turning off the cluster again.

[TomNicholas]

More information here

https://docs.coiled.io/user_guide/prefect.html

Note that you can do this without Docker, as it works with Coiled's Package Sync. The spin-up time is said to be ~30-60s.

There are now two interesting possibilities for running Cubed on Coiled without Dask* which makes Coiled much more similar to Modal for our deployment purposes.

*The other being to run with the single-machine executor on a single fat instance that you requisition via a Coiled notebook.

[jacobtomlinson]

Cubed might be a fun use case for Coiled's batch jobs too. SInce they announced it I've been trying to think of a practical use case for it.