LightningCLI: suggested structure for callbacks/loggers

[adamjstewart]

In microsoft/torchgeo#1559, we're trying to replace our custom train.py script with a LightningCLI implementation. There are still a couple remaining things that I'm trying to translate.

By default, we would like all runs to use the following callbacks:

• ModelCheckpoint
• EarlyStopping

and loggers:

• CSVLogger
• TensorBoardLogger

We would also like the user to be able to control where the checkpoints/log files get saved (experiment directory) and what name they are saved under (experiment name).

ModelCheckpoint has an optional dirpath attribute. If it isn't set, the user can control it via --trainer.default_root_dir. However, the save_dir parameter of CSVLogger and TensorBoardLogger is required, so it doesn't seem possible to use --trainer.default_root_dir to control it. And the experiment name has to be passed when creating the logger. What's the best way to handle this?

@calebrob6

[mauvilsa]

we would like all runs to use the following callbacks

For this you should configure-forced-callbacks.

For the loggers I am not sure.

[adamjstewart]

We put our callbacks directly in the LightningModule using configure_callbacks: https://github.com/microsoft/torchgeo/pull/1559/files#diff-16e0fe9e6a17ad70cde11f15902984231f9223a0333a5d42cf8e49e2e53bb562R39

Is that bad? That's what was recommended in #18480.

[mauvilsa]

No, that is not bad. The forced callbacks is when you want a callback to be configurable in the CLI similar to other classes.

[jxtngx]

@adamjstewart Just curious if you still need clarification; I noticed the linked PR has been merged.

[adamjstewart]

Still looking for suggestions. We haven't yet added logging. Would others be interested in a PR that lets our loggers use trainer.default_root_dir just like ModelCheckpoint if no save_dir is specified?