Title: Bimanual gesture recognition based on convolution neural network

Authors: Hao Wu; Gongfa Li; Ying Sun; Guozhang Jiang; Du Jiang

Addresses: Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Wuhan University of Science and Technology, Wuhan, Hubei, China; Research Centre for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, Hubei, China ' Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, Hubei, China; Research Centre for Biomimetic Robot and Intelligent Measurement and Control, Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, Hubei, China ' Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, Hubei, China ' Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, Hubei, China; Hubei Key Laboratory of Mechanical Transmission and Manufacturing Engineering, Wuhan University of Science and Technology, Wuhan, Hubei, China ' Key Laboratory of Metallurgical Equipment and Control Technology of Ministry of Education, Institute of Precision Manufacturing, Wuhan University of Science and Technology, Wuhan, Hubei, China; Research Centre for Biomimetic Robot and Intelligent Measurement and Control, Wuhan University of Science and Technology, Wuhan, Hubei, China

Abstract: Gesture recognition is a key research field in the human-computer interaction. At present, most of researchers focus on one-handed gesture recognition, but do not pay much attention to bimanual (two hands) gesture recognition. This paper presents a deep learning-based solution to tackle the self-occlusion and self-similarity. To solve this problem, this paper uses Kinect to collect many colour and depth images of different gestures, and each gesture contains multiple sample individuals. Colour images and depth images are used to train the recognition model of bimanual gesture respectively, and then the colour image and depth image are fused, and the bimanual gesture recognition model is trained based on colour image and depth image fusion. Then, the bimanual recognition effects of the three models are compared. The experimental results show that, regardless of the single gesture precision or the mean average precision, the bimanual gesture recognition effect of the fused model is better than the gesture recognition models based on either colour image or depth image.

Keywords: gesture recognition; bimanual gesture; deep learning; CNN; occlusion.

DOI: 10.1504/IJWMC.2020.108523

International Journal of Wireless and Mobile Computing, 2020 Vol.18 No.4, pp.311 - 319

Received: 10 Jul 2019
Accepted: 10 Aug 2019
Published online: 16 Jul 2020 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article