Why have zero initializations for norm layers been removed?

[kami93]

Hi. I noticed that the constant initializations for LayerNorms (ones for the weights, zeros for the biases) in ViT are removed in the latest master. I am wondering about the reason for this change and what is the consequence? I tried to look for clues in commit msgs or changelogs, but I could not figure out any. I really appreciate any help you can provide.

Comparing:
https://github.com/rwightman/pytorch-image-models/blob/372ad5fa0dbeb74dcec81db06e9ff69b3d5a2eb6/timm/models/vision_transformer.py#L379-L403 (the latest master)

vs.

https://github.com/rwightman/pytorch-image-models/blob/v0.5.4/timm/models/vision_transformer.py#L376-L408 (tag v0.5.4)

[rwightman]

@kami93 that's the default init for those layers

[rwightman]

More specifically, this is what's in torch LayerNorm,reset_parameters() is called in __init__() .... this is also true for all of the other norm layers. Up until now I've been cutting and pasting and recyclng code that had that explicitly in the network init, but now startnig to remove them when I touch the code as it's redudnant. The goal now is to only have net inits cover the parts that aren't defaults...

    def reset_parameters(self) -> None:
        if self.elementwise_affine:
            init.ones_(self.weight)
            init.zeros_(self.bias)

NOTE that I have been experimenting and adjusting the weight inits for some nets like vit, although have been trying to keep backwards compatible modes in place but if I find an init that works better across a range of hparams / optimizers, etc I may switch...