Core50, 50 classes.

[DraBard]

Is there an easy way for the new classes scenario with 50 classes to split the experience 0 into two experiences
with 5 different classes each?

With default configuration it has 10 experiences where experience 0 has 10 classes and the rest experiences have 5 classes each.
With such configuration when I set up the early stopping it always triggers on the beginning of training for all of the experience (except experience 0), because the highest average accuracy is achieved on experience 0. And then classes from experience 0 are when the model was saved and it propagates throughout all training. That is because more classes means more contribution to the average.

Am I missing something here, or do I make some mistake in the reasoning ?

[vlomonaco]

This is the method you are looking for: https://avalanche-api.continualai.org/en/v0.3.1/generated/avalanche.benchmarks.generators.nc_benchmark.html#avalanche.benchmarks.generators.nc_benchmark

Starting from the CORe50 dataset you can create any specific split of classes! The rationale of having more classes in the first batch is motivated by the fact that many real-world applications can start from a reasonable dataset before deployment and some methods can leverage that.

[DraBard]

First of all, thank you very much for quick and fruitful answer. Unfortunately, I still struggle with the core50 set.

Having coded the following:

core50 = CORe50(scenario="nc", mini=True, object_lvl=True)
core50 = nc_benchmark(core50.train_stream, core50.test_stream, n_experiences=10, shuffle=True, seed=1234, task_labels=False)

I get the ValueError as below:

Traceback (most recent call last):
  File "/Users/bartlomiejpalus/Documents/Masters-Thesis/train.py", line 2220, in <module>
    core50_check()
  File "/Users/bartlomiejpalus/Documents/Masters-Thesis/train.py", line 877, in core50_check
    core50 = nc_benchmark(core50.train_stream, core50.test_stream, n_experiences=9, shuffle=True, seed=1234, task_labels=False)
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/generators/benchmark_generators.py", line 221, in nc_benchmark
    train_dataset = make_classification_dataset(
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/utils/classification_dataset.py", line 212, in make_classification_dataset
    targets = _init_targets(dataset, targets)
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/utils/classification_dataset.py", line 318, in _init_targets
    targets = _traverse_supported_dataset(dataset, _select_targets)
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/utils/classification_dataset.py", line 900, in _traverse_supported_dataset
    raise initial_error
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/utils/classification_dataset.py", line 836, in _traverse_supported_dataset
    result = values_selector(dataset, indices)
  File "/Users/bartlomiejpalus/miniforge3/envs/avalanche/lib/python3.10/site-packages/avalanche/benchmarks/utils/classification_dataset.py", line 799, in _select_targets
    raise ValueError(
ValueError: Unsupported dataset: must have a valid targets field or has to be a Tensor Dataset with at least 2 Tensors

Is it some dataset bug, or do I do something incorrectly ?

[DraBard]

I just needed to use CORe50Dataset, no the benchmark ... .

[AntonioCarta]

I don't think there is a "CORe50 dataset" since CORe50 is a collection of streams. If you want a joint dataset you can make a class-incremental stream and concatenate all of them together. Then, you can split them how you want.