Different result from train and validation

[chamecall]

I wanted to check training and validation tendency on one fixed batch so I created two equal datasets (train and val) and appropriate dataloaders and called Trainer.fit() method for 40 epochs.
The objective is binary classification. Loss function is binary_crossentropy_with_logits.
In case of training it all looks right cause with every following epoch loss is approximately decreasing and predictions become more and more right but in case of validation ( I would like to remind you that this is the same data which used in the training process) we don't get loss deacrease and we have bad class predictions.

Can you explain what could be the reason of the behaviour?
Below I attached outputs for the 40 epochs and losses plot.

train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.0055, -0.1047], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.6662997007369995
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0191, 0.0705], device='cuda:0')
loss: 0.7063159942626953



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([-0.0050, -0.1425], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.6600629091262817
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0212, 0.0738], device='cuda:0')
loss: 0.7066450119018555



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.0619, -0.1295], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.646587073802948
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0223, 0.0755], device='cuda:0')
loss: 0.706830620765686



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.2458, -0.2895], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.5683196783065796
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0229, 0.0765], device='cuda:0')
loss: 0.7069603204727173



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.3811, -0.3469], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.5276488065719604
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0229, 0.0767], device='cuda:0')
loss: 0.707007884979248



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.3401,  0.3228], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.5410979986190796
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0222, 0.0761], device='cuda:0')
loss: 0.7070177793502808



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.4356, -0.4382], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.4983615577220917
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0219, 0.0759], device='cuda:0')
loss: 0.7070218324661255



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.5919, -0.5687], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.44452834129333496
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0210, 0.0750], device='cuda:0')
loss: 0.7070424556732178



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.5665,  0.4124], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.47877565026283264
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0195, 0.0735], device='cuda:0')
loss: 0.7069916129112244



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.4869,  0.2799], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.5209900736808777
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0179, 0.0717], device='cuda:0')
loss: 0.7069323062896729



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.8491,  0.7699], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.3683297038078308
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0168, 0.0705], device='cuda:0')
loss: 0.7068970203399658



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.5816, -0.6744], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.427901029586792
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0156, 0.0687], device='cuda:0')
loss: 0.706729531288147



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.6160, -0.5898], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.43647289276123047
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0144, 0.0670], device='cuda:0')
loss: 0.7065752744674683



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.5929, -0.5894], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.44061967730522156
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0138, 0.0659], device='cuda:0')
loss: 0.7064590454101562



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.6966,  0.7339], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.39818668365478516
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0132, 0.0647], device='cuda:0')
loss: 0.7063093185424805



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.7265, -0.8689], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.372355580329895
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0129, 0.0637], device='cuda:0')
loss: 0.706124484539032



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.7280, -0.9996], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.35366785526275635
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0130, 0.0632], device='cuda:0')
loss: 0.705959677696228



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.8815,  0.9432], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.3376961946487427
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0136, 0.0634], device='cuda:0')
loss: 0.7058513760566711



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.0051, -1.0752], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.3027455806732178
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0146, 0.0640], device='cuda:0')
loss: 0.7057628631591797



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.8699, -0.8890], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.34714847803115845
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0161, 0.0650], device='cuda:0')
loss: 0.7056518793106079



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.0824, -1.2147], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.2758219838142395
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0184, 0.0670], device='cuda:0')
loss: 0.7055959701538086



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.9776,  1.1215], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.3006746172904968
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0210, 0.0688], device='cuda:0')
loss: 0.7054272890090942



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.1971,  1.2245], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.2608024477958679
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0244, 0.0715], device='cuda:0')
loss: 0.7052828073501587



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.3767,  1.2886], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.23426291346549988
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0268, 0.0731], device='cuda:0')
loss: 0.7051010131835938



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.1933, -1.1563], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.26920002698898315
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0298, 0.0753], device='cuda:0')
loss: 0.7049109935760498



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-0.7806,  0.7893], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.37578916549682617
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0328, 0.0770], device='cuda:0')
loss: 0.7046339511871338



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 0.9963, -1.0034], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.31330639123916626
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0359, 0.0789], device='cuda:0')
loss: 0.7043588161468506



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.2029, -1.2848], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.25344032049179077
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0393, 0.0811], device='cuda:0')
loss: 0.7041041851043701



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.5567, -1.5121], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.19526255130767822
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0437, 0.0841], device='cuda:0')
loss: 0.7038190364837646



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.4261,  1.3994], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.2179250717163086
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0469, 0.0857], device='cuda:0')
loss: 0.7034397125244141



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.5468,  1.4720], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.1998053789138794
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0512, 0.0885], device='cuda:0')
loss: 0.7031115293502808



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.4304,  1.2825], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.22962906956672668
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0554, 0.0912], device='cuda:0')
loss: 0.7027970552444458



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.3031,  1.4736], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.2233087569475174
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0591, 0.0933], device='cuda:0')
loss: 0.7024389505386353



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.5555, -1.6675], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.18219852447509766
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0625, 0.0951], device='cuda:0')
loss: 0.702099621295929



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.6438, -1.9140], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.15713074803352356
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0663, 0.0973], device='cuda:0')
loss: 0.7017720937728882



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.5302, -1.4787], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.2006484568119049
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0703, 0.0998], device='cuda:0')
loss: 0.701443076133728



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.8580,  1.9829], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.13696931302547455
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0753, 0.1031], device='cuda:0')
loss: 0.7011159658432007



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.5302, -1.6989], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.18196645379066467
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0787, 0.1048], device='cuda:0')
loss: 0.7007445096969604



train:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([ 1.7865, -1.7752], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.15571488440036774
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0823, 0.1066], device='cuda:0')
loss: 0.7003462314605713



train:
y: tensor([0., 1.], device='cuda:0')
y_hat: tensor([-1.6773,  1.5574], device='cuda:0', grad_fn=<SqueezeBackward1>)
loss: 0.18125741183757782
val:
y: tensor([1., 0.], device='cuda:0')
y_hat: tensor([0.0857, 0.1082], device='cuda:0')
loss: 0.6999489068984985

[tchaton]

Dear @chamecall,

It would be hard to provide an answer without access to the code.

However, be aware the data for training are randomly shuffled.

Best,
T.C