course 78 where the activation layer put?

[chos1npc]

• Course video/timestamp: (video number 78)

My error

TODO: I try to add the sigmoid to the model on bottom, I think it can work, but it didn't.

My code

Build a model with non-linear activation functions

from torch import nn
class CircleModelV2(nn.Module):
def init(self):
super().init()
self.layer_1 = nn.Linear(in_features=2, out_features=10)
self.layer_2 = nn.Linear(in_features=10, out_features=10)
self.layer_3 = nn.Linear(in_features=10, out_features=1)
self.relu = nn.ReLU() # relu is a non-linear activation function
self.sigmoid = nn.Sigmoid()

def forward(self, x):
# Where should we put our non-linear activation functions?
return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))
return self.sigmoid(self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x))))))

model_3 = CircleModelV2().to(device)

Random seeds

torch.manual_seed(42)
torch.cuda.manual_seed(42)

Put all data on target device

X_train, y_train = X_train.to(device), y_train.to(device)
X_test, y_test = X_test.to(device), y_test.to(device)

Loop through data

epochs = 1000

for epoch in range(epochs):

Training

model_3.train()

1. Forward pass

y_logits = model_3(X_train).squeeze()

y_pred = torch.round(torch.sigmoid(y_logits)) # logits -> prediction probabilities -> prediction labels

y_pred = y_logits

2. Calculate the loss

loss = loss_fn(y_logits, y_train) # BCEWithLogitsLoss (takes in logits as first input)
acc = accuracy_fn(y_true=y_train,
y_pred=y_pred)

3. Optimizer zero grad

optimizer.zero_grad()

4. Loss backward

loss.backward()

5. Step the optimizer

optimizer.step()

Testing

model_3.eval()
with torch.inference_mode():
test_logits = model_3(X_test).squeeze()
test_pred = test_logits
# test_pred = torch.round(torch.sigmoid(test_logits))

test_loss = loss_fn(test_logits, y_test)
test_acc = accuracy_fn(y_true=y_test, 
                       y_pred=test_pred)

Print out what's this happenin'

if epoch % 100 == 0:
print(f"Epoch: {epoch} | Loss: {loss:.4f}, Acc: {acc:.2f}% | Test Loss: {test_loss:.4f}, Test Acc: {test_acc:.2f}%")

[BooBoa]

Hi chos1npc,

nn.BCEWithLogitsLoss function expects raw logits as it's input to calculate the loss.
With a sigmoid activation layer in your model, your model is outputting prediction probabilities which is then being fed into your loss function but it requires raw logits.

Our model should look similiar to this the nn.ReLU() is our non linear layer

from torch import nn

class CircleModelV3(nn.Module):
  def __init__(self) -> None:
    super().__init__()
    self.layer_1 = nn.Linear(in_features=2, out_features=10)
    self.layer_2 = nn.Linear(in_features=10, out_features=10)
    self.layer_3 = nn.Linear(in_features=10, out_features=1)
    self.relu = nn.ReLU()

  def forward(self, x: torch.Tensor) -> torch.Tensor:
    # Put our relu function inbetween our layers
    return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))
     ```
Our training loop should be similiar to this, we pass our logits to the loss function to calculate the loss,
then we call the sigmoid and round function  on our logits to pass to our accuracy function.
```# Build loop

for epoch in range(epochs):
  model_3.train()
  y_logits = model_3(X_train).squeeze()
  y_preds = torch.round(torch.sigmoid(y_logits))
  loss = loss_fn(y_logits,
                 y_train)
  acc = accuracy_fn(y_true=y_train,
                    y_pred=y_preds)
  optimizer.zero_grad()
  loss.backward()
  optimizer.step()

  model_3.eval()
  with torch.inference_mode():
    test_logits = model_3(X_test).squeeze()
    test_preds = torch.round(torch.sigmoid(test_logits))
    test_loss = loss_fn(test_logits,
                        y_test)
    test_acc = accuracy_fn(y_true=y_test,
                           y_pred=test_preds)```



A kind fisherman
 

[chos1npc]

so we need raw logits to calculate the loss?
but why many model can use in classfication, it's output label. eg.resnet

[BooBoa]

yes the loss function you are using in that part of the course requires logits but try out diferrent loss functions, im just a little further ahead than you in the course image classification is next :)
i did have some fun trying to replicate your model with the sigmoid layer, how i worked through it is here if your interested

https://github.com/BooBoa/school/blob/main/Sigmoid_activation_layer.ipynb

[chos1npc]

I think that I understand, thanks

[nortln]

Hmmm