Train multiple models in a single GPU on parallel

[grudloff]

Is there a recommended way of training multiple models in parallel in a single GPU? I tried using joblib's Parallel & delayed but I got a CUDA OOM with two instances even though a single model uses barely a fourth of the total memory. And is a speedup compared to sequential calling expected?

[Programmer-RD-AI]

Solution https://pytorch-lightning.readthedocs.io/en/stable/advanced/multi_gpu.html

[Programmer-RD-AI]

Solution #2807

[Programmer-RD-AI]

Solution https://wandb.ai/wandb/wandb-lightning/reports/Multi-GPU-Training-Using-PyTorch-Lightning--VmlldzozMTk3NTk

[grudloff]

Thanks, @Programmer-RD-AI. Sadly your solutions point to multi-gpu training, but I am looking into training multiple models on a single GPU (and not the other way around!). And the issue you reference is just training a lot of models sequentially (if I understood correctly)

[Programmer-RD-AI]

oh, ok

[awaelchli]

@grudloff I don't know how this could be done or if this is even possible, but if you don't find the answer here perhaps you could also ask in the PyTorch forum.

[cgerum]

It is possible to run multiple trainings on a single GPU using joblib. But as far as I know, you need to instantiate your Trainer and your LightningModule wether you will see speedups, highly depends on your model. If a single model already has a good GPU utilization, you will not see any speedups.

[rustamzh]

Hi, I am doing testing in parallel on a single GPU using joblib. I am instantiating all modules inside the delayed function. However, I had a problem with exceptions in pytorch (relevant issue), which was solved by specifying multiprocessing_context in DataLoader (relevant issue comment)

[rustamzh]

Regarding the speed up, yes definitely in testing

[grudloff]

could you provide a code snippet?

[rustamzh]

Here are relevant snippets:

        with parallel_backend('loky',n_jobs=n_jobs):
            metrics = Parallel()(delayed(model_testing)(**test_param) for test_param in get_test_params(...))

def model_testing(...):

    datamodule = None
    with warnings.catch_warnings():
        warnings.simplefilter("ignore")
        datamodule = get_datamodule_test(...)

    if not datamodule:
        return None

    trainer = pl.Trainer(
        enable_model_summary=False,
        enable_progress_bar=False,
        logger=False,
        log_every_n_steps=10,
        check_val_every_n_epoch=1,
        enable_checkpointing=True,
        accelerator="auto",
        devices=1,
    )

    rows = dict()
    
    test_metrics = trainer.test(model, datamodule=datamodule, verbose=False)
    rows[(...)] = test_metrics[0]
    
    return rows