Title: Deep neural networks for multimodal data fusion and affect recognition

Authors: Dhruv Bhandari; Sandeep Paul; Apurva Narayan

Addresses: Dayalbagh Educational Institute, Dayalbagh, Agra, Uttar Pradesh – 282005, India ' Dayalbagh Educational Institute, Dayalbagh, Agra, Uttar Pradesh – 282005, India ' Department of Computer Science, The University of British Columbia, Kelowna, BC V1Y 1V7, Canada; Department of Systems Design Engineering, University of Waterloo, 200 University Ave W, Waterloo, ON N2L 3G1, Canada

Abstract: This paper proposes novel deep neural network models to handle multimodal data. The proposed models seamlessly facilitate fusion of multimodal inputs and bring about dimensional reduction of the input feature space. The architecture employs multimodal stacked autoencoder in conjunction with multi-layer perceptron-based regression model. Two variants of the architecture are proposed. Experiments have been performed on the multimodal benchmark dataset (RECOLA) to illustrate the importance of multimodality for affect recognition. The proposed architectures are trained using effective training strategies, specifically designed to reduce the number of tuneable parameters for multimodal applications. The results obtained are encouraging and the proposed approach is computationally less expensive than the existing approaches. The performance is better or at par with the other techniques.

Keywords: multimodal data; stacked autoencoder; SAE; deep neural network; DNN; data fusion; affect recognition; emotional recognition; multimodal stacked autoencoder; MSAE.

DOI: 10.1504/IJAISC.2020.113475

International Journal of Artificial Intelligence and Soft Computing, 2020 Vol.7 No.2, pp.130 - 145

Received: 04 Apr 2018
Accepted: 14 Apr 2019
Published online: 08 Mar 2021 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article