Step logs access from the Callback for more than once logging during a single epoch.

[DomainFlag]

Hi, how is it possible to have logs access (metrics) from the Callback for more than once logging during a single epoch? As of now, I am using the pytorch_lightning.callbacks where using the callback_metrics whenever and epoch (train/validation) ends by managing multiple progress bars. But as the complexity increases and the number of batches have increased too, it would be very handy for me to have logging not only epoch wise, but also batch wise where all the metrics are aggregated based on some print_every that is less than the number of batches used for training.

How can I do it? I am looking in the worst-case scenario for a variable maybe inside Trainer that has the list of all the step-wise metrics, that I can tackle during the callback call on_train_batch_end?

[tchaton]

Dear @DomainFlag,

I am not sure to fully understand what you are trying to do. Mind sharing a pseudo code of your use-case.

The metrics are stored within ResultCollection objects stored for each loop.

You can access one for example by trainer.fit_loop.epoch_loop._results.

[DomainFlag]

@tchaton Hi! Sorry, for the late answer. Unfortunately, trainer.fit_loop.epoch_loop._results is not available as I'm using the "1.2.8" version.

What I'm trying to do, is that there is the abstract class Callback that I'm implementing (e.g. on_train_epoch_end) where trainer.callback_metrics are exposed with epoch aggregated metrics (mean over all batches). It does the job well, but if the number of batches are way too high, conversely the logger will output epoch metrics after a long time + the step metrics aren't much useful either as they vary too often. I am looking for a way to log the aggregated step metrics over a smaller subset of batches used for training to have faster progress results than having to wait the whole epoch and process every batch.

(E.g.) Let's say I have 200 000 samples with a batch size of 24 that's 8 333 batches, if every batch takes 2s to process, then a whole epoch will take 8 333 * 2 ~ 4 hours which is a lot. I am looking for some type of mini-epoch logging (log every 1000 batches), where all the logs are either aggregated (mean) over a span of exactly 1 000 batches or from the start of epoch training ([0, 1000], [0, 2000], [0, 3000], ....), either of those two. That will be very helpful for prototyping and experimenting to better know if something goes south (wrong).

I really don't know what's the best way to do it, but what I am doing as of now is implementing the on_train_batch_end(self, trainer, pl_module, outputs, batch, batch_idx, dataloader_idx) of the abstract class Callback and use the outputs with the attribute extra to manually aggregate in online fashion. I am wondering if there are any better ways to do that?

Note: I would also not want to upgrade the package, as the API changes way too often and I am always forced to make some custom old features compatible again with the newer version.