Automatic Sentiment Labelling of Multimodal Data

This study investigates the challenging problem of automatically providing sentiment labels for training and testing multimodal data containing both image and textual information for supervised machine learning. Because both the image and text components, individually and collectively, convey sentiment, assessing the sentiment of multimodal data typically requires both image and text information. Consequently, the majority of studies classify sentiment by combining image and text features (‘Image+Text-features’). In this study, we propose ‘Combined-Text-Features’ that incorporate the object names and attributes identified in an image, as well as any accompanying superimposed or captioned text of that image, and utilize these text features to classify the sentiment of multimodal data. Inspired by our prior research, we employ the Afinn labelling method to automatically provide sentiment labels to the ‘Combined-Text-Features’. We test whether classifier models, using these ‘Combined-Text-Features’ with the Afinn labelling, can provide comparable results as when using other multimodal features and other labelling (human labelling). CNN, BiLSTM, and BERT models are used for the experiments on two multimodal datasets. The experimental results demonstrate the usefulness of the ‘Combined-Text-Features’ as a representation for multimodal data for the sentiment classification task. The results also suggest that the Afinn labelling approach can be a feasible alternative to human labelling for providing sentiment labels.