Title: Deep fusion of multi-channel neurophysiological signal for emotion recognition and monitoring

Authors: Xiang Li; Dawei Song; Peng Zhang; Yuexian Hou; Bin Hu

Addresses: Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China ' Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China; School of Computing and Communications, The Open University, Milton Keynes MK76AA, UK ' Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China ' Tianjin Key Laboratory of Cognitive Computing and Application, School of Computer Science and Technology, Tianjin University, Tianjin 300350, China ' The Ubiquitous Awareness and Intelligent Solutions Lab, School of Information Science and Engineering, Lanzhou University, Lanzhou 730000, China

Abstract: How to fuse multi-channel neurophysiological signals for emotion recognition is emerging as a hot research topic in community of Computational Psychophysiology. Nevertheless, prior feature engineering based approaches require extracting various domain knowledge related features at a high time cost. Moreover, traditional fusion method cannot fully utilise correlation information between different channels and frequency components. In this paper, we design a hybrid deep learning model, in which the 'Convolutional Neural Network (CNN)' is utilised for extracting task-related features, as well as mining inter-channel and inter-frequency correlation, besides, the 'Recurrent Neural Network (RNN)' is concatenated for integrating contextual information from the frame cube sequence. Experiments are carried out in a trial-level emotion recognition task, on the DEAP benchmarking dataset. Experimental results demonstrate that the proposed framework outperforms the classical methods, with regard to both of the emotional dimensions of Valence and Arousal.

Keywords: affective computing; CNN; time series data analysis; EEG; emotion recognition; LSTM; multi-channel data fusion; multi-modal data fusion; physiological signal; RNN.

DOI: 10.1504/IJDMB.2017.086097

International Journal of Data Mining and Bioinformatics, 2017 Vol.18 No.1, pp.1 - 27

Received: 27 Mar 2017
Accepted: 03 May 2017
Published online: 24 Aug 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article