Title: A reinforcement learning-based multimodal scenario hazardous behaviour recognition method

Authors: Di Sun; Yanjing Li; Yuexia Han

Addresses: Department of Information Engineering, Shijiazhuang University of Applied Technology, Shijiazhuang 050081, China ' Department of Information Engineering, Shijiazhuang University of Applied Technology, Shijiazhuang 050081, China ' Department of Information Engineering, Shijiazhuang University of Applied Technology, Shijiazhuang 050081, China

Abstract: In order to solve the problems of low recognition accuracy, low accuracy of key frame extraction of dangerous behaviours, and long delay time in risk behaviour recognition methods in multimodal scenes, this paper studies a multimodal scene risk behaviour recognition method based on reinforcement learning. Firstly, static and dynamic dangerous behaviour features are extracted through long short-term memory artificial neural network. Then, the extracted dangerous behaviour key frame set is optimised by reinforcement learning. Finally, a multi-sensory risk behaviour model library is constructed to complete the multi-modal scene dangerous behaviour recognition. Experiments show that the AUC area of the proposed method is closest to 1, which proves that the recognition accuracy is high, and the accuracy of the extracted key frames of dangerous behaviours is high. When the maximum number of video frames is 500 frames, the delay time of this method is 192 ms, and the shorter the delay time; the shortest, the higher the recognition efficiency.

Keywords: reinforcement learning; multimodal scene; hazardous behaviour recognition; neural network.

DOI: 10.1504/IJCISTUDIES.2023.132489

International Journal of Computational Intelligence Studies, 2023 Vol.12 No.1/2, pp.52 - 71

Received: 29 Sep 2022
Accepted: 01 Dec 2022
Published online: 24 Jul 2023 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article