Title: Metaverse-driven instrumental music teaching: a human-computer interactive perception method

Authors: Tianyu Chen

Addresses: Sejong University, Seoul, South Korea

Abstract: Teaching musical instruments traditionally requires one-on-one instruction, creating challenges of accessibility, cost and feedback quality. Metaverse technologies offer promising solutions but face limitations in gesture recognition accuracy and haptic feedback realism. This paper proposes a metaverse-driven human-computer interaction method for instrumental music teaching that addresses these challenges. Through analysis of similar gestures in instrumental music performance, we discovered that inflection point count and gesture displacement vector angle significantly improve discrimination between similar instrumental gestures. By incorporating these features into our dynamic gesture recognition method, we enhance identification accuracy for specialised musical movements. Additionally, we simulate the haptic interaction process between virtual hands and instruments to achieve realistic performance effects and synchronous multi-channel sensory feedback in real-time interactions. Our approach focuses on six degrees of freedom haptic interactions during multi-point contact scenarios. Experimental results demonstrate that our method achieves higher recognition accuracy than baseline approaches for customised gestures and instrument-specific movements. This research contributes to more accessible, effective and immersive instrumental music education in virtual environments, potentially democratising access to quality music instruction regardless of geographical or economic constraints.

Keywords: metaverse; instrumental teaching; human-computer interaction; VR; leap motion; six degrees of freedom.

DOI: 10.1504/IJCAT.2025.149369

International Journal of Computer Applications in Technology, 2025 Vol.76 No.3/4, pp.207 - 221

Received: 11 May 2024
Accepted: 20 Mar 2025
Published online: 27 Oct 2025 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article