ResNet strikes back procedure D

[omerb01]

Hey, I'm looking for a train script for procedure D in the "ResNet strikes back" paper: https://arxiv.org/pdf/2110.00476.pdf
it's a bit confusing because there are some hyper parameters that are not mentioned regarding default arguments in timm repo, and there are some arguments that are mentioned in procedures A1 A2 A3 but not in the others.

I would be happy for help here,
thanks, Omer.

[rwightman]

@omerb01 between table 10 (D column) and table 13 I think everything is covered and fairly clear relative to timm train script args? I have a yaml config file for that specific run but it's on a remote machine that's being ugpraded and not on at the moment.

WRT to the timm train scripts, for the most part the defaults can be ignored and you should only need to set the hparams that are explicitly mentioned in the paper.

One omission relative to train.py defaults is --warmup-lr 1e-6

I think it'd be pretty close to this (run on a 2x 2080Ti machine, so most low-end machine of all paper experiments)
./distributed_train.sh 2 /imagenet --model resnet50 --amp --batch 192 --lr 3.3e-3 --epochs 600 --weight-decay 0.01 --sched cosine --train-interpolation bicubic --crop-pct 0.95 --smoothing 0.1 --warmup-epochs 5 --aa rand-m7-n3-mstd1.0-inc1 --seed 0 --opt adamp --warmup-lr 1e-6 --drop-path 0.05 --drop 0.1 --reprob 0.35 --mixup .2 --cutmix 1.0 --bce-loss

One thing that I need to check the config to confirm is if for D it's:
--bce-target-thresh 0.2 --smoothing 0. or --smoothing 0.1 and no bce-target enabled.

[omerb01]

@rwightman Thanks a lot!
not sure if its necessary but when it is possible, it would be great to get this config file so we can be sure about the last setting you mentioned.

[MichaelMonashev]

Weight decay 0.01 looks too big for AdamP, but works good for me.
AdamP produce little norm for weights. Its authors gets better results without WD ( https://arxiv.org/pdf/2006.08217.pdf Table 1. ) So it is unclear, why so big WD=0.01 did not interferes with learning.

[kevindoran]

@rwightman, any chance of getting the config file?