Multiple types of input into a deep learning model?

[solarslurpi]

A slide in the video shows an audio and a (Twitter text) text and an image being converted into a tensor to be provided as input into a deep learning model.

Yet, it seems example use one type of data. E.g.: image classification, NLP...etc.

Thus, it is unclear to me that there is a model that could be fit with both text and image? For example, say I wanted deep learning to determine if a plant has a Nitrogen deficiency. The text might be "overall yellowing of the lower leaves fairly rapidly getting yellow throughout the plant" and there could be a few images with a plant in this state.

How would I use both the image and the text with the effort to improve labeling? Thank you.

[DavS123]

Hi solarslurpi,

I am not Daniel Bourke however I can give you maybe an example how you can achieve your goal (from my naive point of view).
The "easiest" example of such a "model" is probably something like this.
model layers:

1. CNN layers /block (You should probably use transfer learning and use something like EFFNETB0)
2. Transformer/ LSTM etc. (Again you should probably use transfer learning)
3. Flatten layer
4. 4. Classifier (output_size = 2)

Now how can be combine the information???
I would probably do this during the forward pass (similar to what we have learned during the Transfer learning module)
For example (pure pseudo):

def forward(self, x,y):
image_infos = torch.select(input, dim, index) depending on your data
text_info = torch.select(input, dim, index) depending on your data
then you can use the CNN layer for your image data
image_infos_after_cnn = self.CNN_layers(image_infos) (but please turn of the gradient if you use transfer learning)
text_info_after_transformer = self.transformer(text_info) (but please turn of the gradient if you use transfer learning)
(then you can combine the using something like )
combined_text_and_image = torch.cat(image_infos_after_cnn, text_info_after_transformer, dim = 0)
Now you have both information in one tensor!! Finally, you can use your flatten layer and the classifier layer to classify if a plant has a Nitrogen deficiency.

return self.classifier( self.flatten(combined_text_and_image, start_dim = suitable, end_dim = suitable))

Additional remarks:

1. This probably a way to simple model, but I hope this gives you an idea how to combine different types of inputs and how to combine different types of inputs using torch.cat().
2. One simple improvement to this model that I can give you is that you should probably use some batch normalization before you use torch.cat()

PS: I hope this was helpful.

[solarslurpi]

Thank you for your reply. It points me in a direction that at this point makes sense. I have a lot to learn!