ModelCheckpoint seems not work correctly with check_val_every_n_epoch>1

[Mo-Junyang]

I have a callback:

checkpoint_callback = ModelCheckpoint(
        dirpath=ckpt_dir,
        filename="checkpoint",
        every_n_epochs=2
    )

and a Trainer

trainer = pl.Trainer(
        # default_root_dir=root_dir,
        accelerator='gpu',
        devices=[config.cuda_idx],
        max_epochs=200,
        check_val_every_n_epoch=5,
        logger=tb_logger,
        enable_checkpointing=True,
        log_every_n_steps=1,
        callbacks=[best_callback, checkpoint_callback],
        deterministic="warn",
        enable_progress_bar=False,
        # =====dev option=====
        num_sanity_val_steps=0,
        # fast_dev_run=1,
        # limit_train_batches=1,
        limit_val_batches=10
        # limit_train_batches=300,
        # profiler=profiler,

    )

I want to save the checkpoint every 2 epoch, and it does work when check_val_every_n_epoch in Trainer is set to 1.
But now I want to set check_val_every_n_epoch to 5, and still save the checkpoint every 2 epoch.
And I got nothing in my checkpoint folder after 5 epoch (before validation).

[Mo-Junyang]

flag: save_on_train_epoch_end=True. Problem solved.