Title: A survey of multimodal emotion recognition: fusion techniques, datasets, challenges and future directions

Authors: Kuei-Chung Chang; Sheng-Quan Chen

Addresses: International Graduate Institute of Artificial Intelligence, National Yunlin University of Science and Technology, Taiwan ' International Graduate Institute of Artificial Intelligence, National Yunlin University of Science and Technology, Taiwan

Abstract: Emotion recognition is crucial in enhancing the quality of human-computer interaction, education, healthcare, and transportation safety. By integrating data from different sources, multimodal emotion recognition can capture complex emotional signals more comprehensively and accurately than single-modal data sources. This paper comprehensively reviews AI-based multimodal emotion recognition systems, covering deep learning techniques, datasets, challenges, and future research directions. We also present current research techniques, including exploring fusion strategies for different modal data and diverse data fusion methods. Several challenges in multimodal emotion recognition are discussed in this paper, such as the incompleteness of modal data, inconsistency in signal quality, and insufficient model interpretability. The paper points out that future research needs to further explore how to effectively integrate data from different modalities and enhance the adaptability and interpretability of models in practical applications.

Keywords: emotion recognition; multimodal learning; artificial intelligence; feature fusion.

DOI: 10.1504/IJBM.2025.148281

International Journal of Biometrics, 2025 Vol.17 No.5, pp.485 - 510

Received: 27 Aug 2024
Accepted: 19 Nov 2024
Published online: 01 Sep 2025 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article