Issue with EWC Plugin on SMNIST Benchmark

[rch-huang]

🐛 Describe the bug
When using the EWCPlugin with non-multi-head models, such as MLP, on the SMNIST benchmark, the accuracy of learned experiences like Exp000, Exp001 are almost 0.00 or very low, with only the current experience being okay. This issue does not occur with the PMNIST benchmark or when using a multi-head MLP within the SMNIST benchmark.

When using the official example provided in the repository ewc_mnist.py, this problem persists.

BTW, I am curious why the GEM plugin works with MLP models on the SMNIST benchmark. Could this be because EWC inherently supports domain incremental learning (e.g., Permuted MNIST) better than class incremental learning (e.g., Split MNIST)?

🐜 To Reproduce
python3 ewc_mnist.py --scenario=smnist --ewc_mode=separate

🐞 Screenshots
'''

Eval on experience 0 (Task 0) from test stream ended.
ExperienceBWT/eval_phase/test_stream/Task000/Exp000 = -0.9424
ExperienceForgetting/eval_phase/test_stream/Task000/Exp000 = 0.9424
Loss_Exp/eval_phase/test_stream/Task000/Exp000 = 2.7904
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp000 = 0.0000

Eval on experience 1 (Task 0) from test stream ended.
ExperienceBWT/eval_phase/test_stream/Task000/Exp001 = -0.7827
ExperienceForgetting/eval_phase/test_stream/Task000/Exp001 = 0.7827
Loss_Exp/eval_phase/test_stream/Task000/Exp001 = 2.6221
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp001 = 0.0000

Eval on experience 2 (Task 0) from test stream ended.
ExperienceBWT/eval_phase/test_stream/Task000/Exp002 = -0.9756
ExperienceForgetting/eval_phase/test_stream/Task000/Exp002 = 0.9756
Loss_Exp/eval_phase/test_stream/Task000/Exp002 = 2.2203
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp002 = 0.0000

Eval on experience 3 (Task 0) from test stream ended.
ExperienceBWT/eval_phase/test_stream/Task000/Exp003 = -0.7346
ExperienceForgetting/eval_phase/test_stream/Task000/Exp003 = 0.7346
Loss_Exp/eval_phase/test_stream/Task000/Exp003 = 1.8450
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp003 = 0.2432

Eval on experience 4 (Task 0) from test stream ended.
Loss_Exp/eval_phase/test_stream/Task000/Exp004 = 1.2222
Top1_Acc_Exp/eval_phase/test_stream/Task000/Exp004 = 0.9739
'''