Why is my training loss and testing loss not changing? #455
Replies: 4 comments
-
|
The issue is where you have your if statement set up. `model_1.eval() if epoch % 100 == 0:` SHOULD BE `if epoch % 100 == 0: Ensure that they are indented properly or else the code will not run |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hey. # An epoch is one loop through the data
epochs = 300
torch.manual_seed(42)
# Track different values
epoch_count = []
loss_values = []
test_loss_values = []
### Training
# 0. Loop through the data
for epoch in range(epochs):
# Set the model to training mode
model_0.train() # train model in PyTorch sets all parameters that require gradients
# 1. Forward pass
y_pred = model_0(X_train)
# 2. Calculate the lodd
loss = loss_fn(y_pred, y_train)
# print(f'Loss: {loss}')
# 3. Optimizer zero grad
optimizer.zero_grad()
# 4. Perform backpropagation on the loss with respect to the parameters of the model
loss.backward()
# 5. Step the optimizer (perform gradient descent)
optimizer.step() # by default how the optimizer changes will accumulate through the loop so.. we have to zero them above in step 3
model_0.eval() # turns off different settings not needed for the evaluation of the model
# Print out model state_dict()
# print(model_0.state_dict())
### Testing
with torch.inference_mode(): # turns off gradient tracking
# 1. Forward pass
test_preds = model_0(X_test)
# 2. Calculate the loss
test_loss = loss_fn(test_preds, y_test)
# print out what's happening
if epoch % 10 == 0:
epoch_count.append(epoch)
loss_values.append(loss)
test_loss_values.append(test_loss)
print(f'Epoch: {epoch} | Test Loss: {test_loss}') |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hi @BCBeeravelly , In your first example, you are printing out the model's loss every ~100 epochs. It appears that your loss doesn't change because it gets so small very quickly. So it reaches the bottom value after the first 100 epochs and then stops improving from there (because it's already very good). The reason your second piece of code prints a decreasing loss value is because you print it out every 10 epochs (small enough to see the loss decreasing. Try this example (printing out every 1 epoch) to see the loss going down (notebook link): import torch
from torch import nn
# Create weight and bias
weight = 0.7
bias = 0.3
# Create range values
start = 0
end = 1
step = 0.02
# Create X and y (features and labels)
X = torch.arange(start, end, step).unsqueeze(dim=1) # without unsqueeze, errors will happen later on (shapes within linear layers)
y = weight * X + bias
# Split data
train_split = int(0.8 * len(X))
X_train, y_train = X[:train_split], y[:train_split]
X_test, y_test = X[train_split:], y[train_split:]
print(f"Length X_train: {len(X_train)}, Length X_test: {len(X_test)}, Length y_train: {len(y_train)}, Length y_test: {len(y_test)}")
# Setup device
device = "cuda" if torch.cuda.is_available() else "cpu"
# Subclass nn.Module to make our model
class LinearRegressionModelV2(nn.Module):
def __init__(self):
super().__init__()
# Use nn.Linear() for creating the model parameters
self.linear_layer = nn.Linear(in_features=1,
out_features=1)
# Define the forward computation (input data x flows through nn.Linear())
def forward(self, x: torch.Tensor) -> torch.Tensor:
return self.linear_layer(x)
# Set the manual seed when creating the model (this isn't always need but is used for demonstrative purposes, try commenting it out and seeing what happens)
torch.manual_seed(42)
model_1 = LinearRegressionModelV2()
model_1, model_1.state_dict()
# Create loss function
loss_fn = nn.L1Loss()
# Create optimizer
optimizer = torch.optim.SGD(params=model_1.parameters(), # optimize newly created model's parameters
lr=0.01)
# Training
torch.manual_seed(42)
epochs = 100
X_train = X_train.to(device)
X_test = X_test.to(device)
y_train = y_train.to(device)
y_test = y_test.to(device)
for epoch in range(0, epochs):
model_1.train()
# 1. Forward pass
y_pred = model_1(X_train)
#2. Calculate the loss
loss = loss_fn(y_pred, y_train)
# 3. Optimizer zero grad
optimizer.zero_grad()
# 4. Loss backward
loss.backward()
# 5. Optimizer step
optimizer.step()
### Testing
model_1.eval()
with torch.inference_mode():
test_preds = model_1(X_test)
test_loss = loss_fn(test_preds, y_test)
if epoch % 1 == 0:
print(f'Epoch: {epoch} | Train loss: {loss} | Test loss: {test_loss}')Output: |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
Hey all, |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi guys, I was trying to work on the LinearRegressionModelv2 in the Section 3 - PyTorch Workflow, but for some reason, my model is not training (aka the training loss and test loss remain the same and the weight and bias do not change during training). Please tell me if I am missing out on something
Lets write a training loop
Beta Was this translation helpful? Give feedback.
All reactions