Loss not decreasing at all - issue when re-initializing model

[chrisBioInf]

Just to make sure I am not crazy and the only one who stumbled here - I am in module 3 (Workflow) and I could, for the love of me, not get the loss value to actually decrease. Parameters, i.e. weight and bias, were also completely static, no matter how many epochs I run.

To check, I literally just copied the code:

torch.manual_seed(42)

# Set the number of epochs (how many times the model will pass over the training data)
epochs = 100

# Create empty loss lists to track values
train_loss_values = []
test_loss_values = []
epoch_count = []

for epoch in range(epochs):
    ### Training

    # Put model in training mode (this is the default state of a model)
    model_0.train()

    # 1. Forward pass on train data using the forward() method inside 
    y_pred = model_0(X_train)
    # print(y_pred)

    # 2. Calculate the loss (how different are our models predictions to the ground truth)
    loss = loss_fn(y_pred, y_train)

    # 3. Zero grad of the optimizer
    optimizer.zero_grad()

    # 4. Loss backwards
    loss.backward()

    # 5. Progress the optimizer
    optimizer.step()

    ### Testing

    # Put the model in evaluation mode
    model_0.eval()

    with torch.inference_mode():
      # 1. Forward pass on test data
      test_pred = model_0(X_test)

      # 2. Caculate loss on test data
      test_loss = loss_fn(test_pred, y_test.type(torch.float)) # predictions come in torch.float datatype, so comparisons need to be done with tensors of the same type

      # Print out what's happening
      if epoch % 10 == 0:
            epoch_count.append(epoch)
            train_loss_values.append(loss.detach().numpy())
            test_loss_values.append(test_loss.detach().numpy())
            print(f"Epoch: {epoch} | MAE Train Loss: {loss} | MAE Test Loss: {test_loss} ")

Yet, I get the following output:

Epoch: 0 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 10 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 20 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 30 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 40 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 50 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 60 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 70 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 80 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 
Epoch: 90 | MAE Train Loss: 0.31288138031959534 | MAE Test Loss: 0.4945361614227295 

The issue here is, that I, in a previous cell, accidentally re-initialized my model by
calling model_0 = LinearRegressionModel() again, AFTER setting up my optimizer.

It seems the optimizer gets strictly tied to the model instance, when it's parameters get passed over the first time - in hindsight this makes sense, but I would wish this would be pointed out in the course somewhere, as this is a mistake that can happen very easily and unnoticed.

Just a heads up, as I saw this topic pop up a few times now with no clear solution.