02. PyTorch Neural Network Classification - put sigmoid function in model

[sudokit]

# Can also put sigmoid in the model 
# This would mean you don't need to use it on the predictions
# self.sigmoid = nn.Sigmoid()

from torch import nn
class CircleModelV2(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer_1 = nn.Linear(in_features=2, out_features=100)
        self.layer_2 = nn.Linear(in_features=100, out_features=100)
        self.layer_3 = nn.Linear(in_features=100, out_features=100)
        self.layer_4 = nn.Linear(in_features=100, out_features=1)
        self.relu = nn.ReLU() # <- add in ReLU activation function
        # Can also put sigmoid in the model 
        # This would mean you don't need to use it on the predictions
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
      # Intersperse the ReLU activation function between layers
       #return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))
       x = self.layer_1(x)
       x = self.relu(x)
       x = self.layer_2(x)
       x = self.relu(x)
       x = self.layer_3(x)
       x = self.relu(x)
       x = self.layer_4(x)
       return self.sigmoid(x)

model_3 = CircleModelV2().to(device)
model_3

and then

y_pred = torch.round(y_logits)

produces different results than

from torch import nn
class CircleModelV2(nn.Module):
    def __init__(self):
        super().__init__()
        self.layer_1 = nn.Linear(in_features=2, out_features=100)
        self.layer_2 = nn.Linear(in_features=100, out_features=100)
        self.layer_3 = nn.Linear(in_features=100, out_features=100)
        self.layer_4 = nn.Linear(in_features=100, out_features=1)
        self.relu = nn.ReLU() # <- add in ReLU activation function
        # Can also put sigmoid in the model 
        # This would mean you don't need to use it on the predictions
        self.sigmoid = nn.Sigmoid()

    def forward(self, x):
      # Intersperse the ReLU activation function between layers
       #return self.layer_3(self.relu(self.layer_2(self.relu(self.layer_1(x)))))
       x = self.layer_1(x)
       x = self.relu(x)
       x = self.layer_2(x)
       x = self.relu(x)
       x = self.layer_3(x)
       x = self.relu(x)
       x = self.layer_4(x)
       return x

model_3 = CircleModelV2().to(device)
model_3

and

y_pred = torch.round(torch.sigmoid(y_logits))

Why is that?

[pusapatiakhilraju]

As per my understanding,
The difference is that in the 1st model, if you have Sigmoid during the forward pass, the model will provide a certain set of outputs and the loss is calculated on those outputs.

In the 2nd mode however, the model will do the forward pass without the Sigmoid and leads to different loss. This also changes the back propagation that is performed on the loss and the gradient steps. Hence, the results are different. (In this model, loss at each step is not calculated on the "sigmoid" generated forward pass like the previous model.)

In the 1st model you are using sigmoid only to get the y_pred values and you do not perform any back propagation on that. It's just used to measure the accuracy

[sudokit]

Oooh. It makes a lot more sense now. TY!
Since it was mentioned in the book, how do you add the sigmoid straight to the model?