How to keep lr fixed at first N epoch, and then use scheduler for the rest of training

[Mo-Junyang]

{"optimizer": optimizer,
 "lr_scheduler": {"scheduler": CosineAnnealingLR(optimizer, self.max_epochs, eta_min=self.min_lr),
                  "interval": "epoch"}}

I want to keep lr at 5e-4 before N epoch, and then use CosineAnnealingLR for the rest of training. What should I do?
I know that I can set self.automatic_optimization = False, and then besides the scheduler I have to manual step the optimizer. Is there any way to do auto optimize and manual scheduler?