Title: Fine-grained emotion recognition: fusion of physiological signals and facial expressions on spontaneous emotion corpus

Authors: Feri Setiawan; Aria Ghora Prabono; Sunder Ali Khowaja; Wangsoo Kim; Kyoungsoo Park; Bernardo Nugroho Yahya; Seok-Lyong Lee; Jin Pyo Hong

Addresses: Department of Industrial and Management Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Industrial and Management Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Industrial and Management Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Information and Communication Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Information and Communication Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Industrial and Management Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Industrial and Management Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea ' Department of Information and Communication Engineering, Hankuk University of Foreign Studies, 89, Wangsan-ri, Mohyeon-eup, Yongin-si, Gyeonggi-do 17035, South Korea

Abstract: The recognition of fine-grained emotions (i.e., happiness, sad, etc.) has shown its importance in a real-world implementation. The emotion recognition using physiological signals is a challenging task due to the precision of the labelled data while using facial expressions is less appropriate for the real environment. This work proposes a framework for fusing physiological signals and facial expressions modalities to improve classification performance. The feature-level fusion (FLF) and decision-level fusion (DLF) techniques are explored in this work to recognise seven fine-grained emotions. The performance of the proposed framework is evaluated using 34 subjects' data. Our result shows that the fusion of the multiple modalities could improve the overall accuracy compared to the unimodal system by 11.66% and 13.63% for facial expression and physiological signals, respectively. Our work achieved a 73.23% accuracy for seven emotions which is considerable accuracy for the spontaneous emotion corpus.

Keywords: emotion recognition; low sampling rate; multimodal fusion; spontaneous facial expression.

DOI: 10.1504/IJAHUC.2020.110824

International Journal of Ad Hoc and Ubiquitous Computing, 2020 Vol.35 No.3, pp.162 - 178

Received: 11 Feb 2020
Accepted: 06 Apr 2020
Published online: 29 Oct 2020 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article