How does swa work with lr schedulers

[maxmatical]

Does the lr stay flat once swa callback starts, or does the lr still follow the scheduler? for example if i use a cosine decay schedule over the entire course of training and using the swa callback starting at 0.8 pct of training, will the swa override the lr schedule and keep the lr flat for the last 20% of training?

[bavShehata]

Yes, When the epoch to use SWA is reached, SWALR (SWA scheduler) will override whatever previous scheduler you had.