Training vit_base_patch16_224_in21k on Cifar10

[animesh-007]

Hi, I was trying to train vit_b_16 on cifar10 using timm library. I am currently using the following command with no augmentation.
python train.py ./datasets/cifar10 -d "torch/cifar10" --dataset-download -b=64 --img-size=224 --epochs=50 --color-jitter=0 --amp --sched='cosine' --model-ema --model-ema-decay=0.995 --min-lr=1e-8 --warmup-epochs=3 --train-interpolation=bilinear --aa=v0 --pretrained --lr=2e-4 --model=vit_base_patch16_224_in21k --opt=adam --weight-decay=1e-4 --num-classes 10

I am getting the best accuracy on the 0th epoch only.
Test (EMA): [ 0/156] Time: 0.126 (0.126) Loss: 0.2100 (0.2100) Acc@1: 93.7500 (93.7500) Acc@5: 100.0000 (100.0000) Test (EMA): [ 50/156] Time: 0.072 (0.076) Loss: 0.1656 (0.1602) Acc@1: 98.4375 (97.5797) Acc@5: 100.0000 (99.9081) Test (EMA): [ 100/156] Time: 0.074 (0.075) Loss: 0.1504 (0.1661) Acc@1: 98.4375 (97.4010) Acc@5: 100.0000 (99.8917) Test (EMA): [ 150/156] Time: 0.076 (0.075) Loss: 0.1578 (0.1669) Acc@1: 98.4375 (97.4234) Acc@5: 100.0000 (99.8965) Test (EMA): [ 156/156] Time: 0.020 (0.075) Loss: 0.0986 (0.1665) Acc@1: 100.0000 (97.4500) Acc@5: 100.0000 (99.8900) *** Best metric: 97.77 (epoch 0)

Can anybody guide me on how I can improve the performance?