Article: Multimodal sentiment analysis based on improved correlation representation network Journal: International Journal of Communication Networks and Distributed Systems (IJCNDS) 2024 Vol.30 No.6 pp.679 - 698 Abstract: Multimodal sentiment analysis (MSA) refers to extracting emotional information from language, acoustic, and visual sequences. Due to the gap between multimodal features, previous work did not take full advantage of multimodal correlations, resulting in limited improvement in fusion strategies. To this end, we propose a multimodal correlation representation network (MCRN) to extract multimodal features by a dual output transformer. The first output of the transformer is used by depth canonical correlation analysis (DCCA) to model the correlation between multimodal data. Then the secondary output of the transformer uses the attention mechanism to fuse multi-modality, and the final output of the model is emotional intensity. In addition, in the second output stage, we designed a single-mode output loss to balance the differences between subtasks. Extensive experiments suggest that our model reaches state-of-the-art performance in most of the existing methods on multimodal opinion-level sentiment intensity (MOSI) dataset and multimodal opinion sentiment and emotion intensity (MOSEI) dataset. Inderscience Publishers - linking academia, business and industry through research

Title: Multimodal sentiment analysis based on improved correlation representation network

Authors: Yilihamu Yaermaimaiti; Tianxing Yan; Guohang Zhuang; Tusongjiang Kari

Addresses: School of Electrical Engineering, Xinjiang University, Urumqi, Xinjiang 830017, China ' School of Electrical Engineering, Xinjiang University, Urumqi, Xinjiang 830017, China ' School of Electrical Engineering, Xinjiang University, Urumqi, Xinjiang 830017, China ' School of Electrical Engineering, Xinjiang University, Urumqi, Xinjiang 830017, China

Abstract: Multimodal sentiment analysis (MSA) refers to extracting emotional information from language, acoustic, and visual sequences. Due to the gap between multimodal features, previous work did not take full advantage of multimodal correlations, resulting in limited improvement in fusion strategies. To this end, we propose a multimodal correlation representation network (MCRN) to extract multimodal features by a dual output transformer. The first output of the transformer is used by depth canonical correlation analysis (DCCA) to model the correlation between multimodal data. Then the secondary output of the transformer uses the attention mechanism to fuse multi-modality, and the final output of the model is emotional intensity. In addition, in the second output stage, we designed a single-mode output loss to balance the differences between subtasks. Extensive experiments suggest that our model reaches state-of-the-art performance in most of the existing methods on multimodal opinion-level sentiment intensity (MOSI) dataset and multimodal opinion sentiment and emotion intensity (MOSEI) dataset.

Keywords: emotion recognition; multimodal learning; multimodal representations; attention fusion.

DOI: 10.1504/IJCNDS.2024.141670

International Journal of Communication Networks and Distributed Systems, 2024 Vol.30 No.6, pp.679 - 698

Received: 08 May 2023
Accepted: 31 Oct 2023
Published online: 30 Sep 2024 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article

Title: Multimodal sentiment analysis based on improved correlation representation network

Keep up-to-date