Multiple dataloaders for val/test in the form of Dict in LightningDataModule

[woozch]

Hi.

I am currently developing a multi-task model that evaluates some datasets with different evaluation metrics for each, but it is impossible to configure multiple val/test dataloader in the form of a dictionary(mapping).

https://github.com/PyTorchLightning/pytorch-lightning/blob/2d043857eef55eb327528b70c8b33c680e82fb7b/pytorch_lightning/core/datamodule.py#L232

It seems that configuring the dataloader in the form of Dict in LightningDataModule is only possible with train_dataloader.
Why not make it possible in val_dataloader or test_dataloader? Is there any other reason for this design?

[rohitgr7]

hey @wgchang !

within train_dataloader when a dict is returned, it creates a CombinedLoader, instead of a sequence of dataloaders.
check out this thread here: #5782 (comment)

[rohitgr7]

yep, it's not allowed. It depends how you want to iterate over the multiple dataloaders. Returning a dict is not allowed.
supported configurations for val_dataloaders is mentioned here: https://pytorch-lightning.readthedocs.io/en/latest/guides/data.html#multiple-validation-test-predict-dataloaders

[rohitgr7]

for your use case, I'd suggest using as a list of dataloaders where you get a dataloader_idx inside your validation_step and you can make a distinction there to configure which metric to use for which dataloader.

[woozch]

Thank you. I'll try to write the code in the way you suggested.

[thesofakillers]

yep, it's not allowed

you should make it allowed.

[surya-narayanan]

I agree, why is there a distinction between the expected behavior of training and test steps?