forward_transfer error after eval on last experience

[h-0-0]

Hi, I'm using "forward_transfer_metrics(experience=True, stream=True)" in my evaluation plug-in and after evaluation finishes on the last experience I am getting the following error trace:

Traceback (most recent call last):
  File "/user/work/jd18380/ContinualLearning/debug.py", line 39, in <module>
    cl_strategy.eval(scenario.test_stream)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/templates/base_sgd.py", line 183, in eval
    super().eval(exp_list, **kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/templates/base.py", line 196, in eval
    self._after_eval(**kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/templates/base.py", line 316, in _after_eval
    trigger_plugins(self, "after_eval", **kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/utils.py", line 69, in trigger_plugins
    getattr(p, event)(strategy, **kwargs)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/plugins/evaluation.py", line 222, in fun
    return self._update_metrics_and_loggers(strat, item)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/training/plugins/evaluation.py", line 160, in _update_metrics_and_loggers
    metric_result = getattr(metric, callback)(strategy)
  File "/user/work/jd18380/pyvenv/lib/python3.10/site-packages/avalanche/evaluation/metrics/forward_transfer.py", line 216, in after_eval
    strategy.eval_every > -1
AssertionError: eval every > -1 to compute forward transfer

Clearly something is going wrong after the evaluation finishes, I've managed to recreate the error using the following script:

from avalanche.logging import TextLogger
from avalanche.training.supervised.strategy_wrappers import Naive
from avalanche.evaluation.metrics import forward_transfer_metrics
from torch.nn import CrossEntropyLoss
from avalanche.training.plugins import EvaluationPlugin
from torch.optim import Adam
from avalanche.benchmarks.classic import SplitCIFAR10
from model import VGG16, ResNet18

text_logger = TextLogger(open('debug.txt', 'a'))

eval_plugin = EvaluationPlugin(
    forward_transfer_metrics(experience=True, stream=True),
    loggers=text_logger
)

model = ResNet18(num_classes=10)

cl_strategy = Naive(
    model, Adam(model.parameters(), lr=0.0001),
    criterion = CrossEntropyLoss(), train_epochs=2,
    train_mb_size = 128, eval_mb_size = 128,
    evaluator=eval_plugin,
    device = "cpu"
)

scenario = SplitCIFAR10(n_experiences=5, shuffle=False)

for experience in scenario.train_stream:
    print("Start of experience: ", experience.current_experience)
    print("Current Classes: ", experience.classes_in_this_experience)

    # train returns a dictionary which contains all the metric values
    cl_strategy.train(experience)
    print('Training completed')

    print('Computing accuracy on the whole test set')
    # test also returns a dictionary which contains all the metric values
    cl_strategy.eval(scenario.test_stream)

Any ideas as to what could be wrong here?

[AntonioCarta]

-1 disables the evaluation, which is needed for forward transfer.