How Does GradientAccumulationScheduler behave if steps in an epoch is not evenly divisible by accumulation steps?

[jeffwillette]

I have been trying to answer this question by looking through the Trainer (https://lightning.ai/docs/pytorch/stable/_modules/lightning/pytorch/trainer/trainer.html#Trainer.__init__) implementation, but I have not found the answer yet. Specifically, say that...

• My dataloader and batch size are such that there are 100 iterations of the dataloader per epoch.
• My gradient accumulation scheduler is set to accumulate every 8 steps.
• 100 % 8 = 4 and therefore there will be 4 steps at the end of the epoch which are not enough for an optimizer step.

I think there are two ways this could be handled, possibly:

1. lightning tracks the number of steps somewhere and starts the new epoch. Then accumulates 4 more steps and performs optimization.
2. it reached the end of the batch and throws away the gradient from the last 4 steps and starts the new epoch.

I would assume that option 1 would be the best choice, but I have not been able to verify that this is the behavior. Does anyone know where to verify this?

[TimandXiyu]

My particular task only has 3-5 steps in each epoch and I set batchsize to 2 and accumulation step to 4. So far, the performance seems correct and sometimes better than no accumulation. So I think we can assume the accumulation step counter will not be reset between epoches.
You might be able to verify this in debug mode, e.g., you can watch a particular layer's weight and check how frequent it is updated. This should be easy to do in IDEs like PyCharm.

[TimandXiyu]

Finding the exact source code in lightning's document is also fine, but I guess using debug mode to observe if it works like what you said in point 1 is sufficient.