Article: Real-time error correction system of spoken English based on multimodal transformer-GCN framework Journal: International Journal of Information and Communication Technology (IJICT) 2026 Vol.27 No.50 pp.1 - 18 Abstract: In today's globalised and technology-driven world, improving spoken English is increasingly important. However, traditional automatic speech recognition (ASR) systems often produce outputs with grammatical errors, poor word choices, and pronunciation ambiguities, hindering effective communication. To address this, we propose MTG-ERR, a novel multimodal transformer-GCN framework that integrates acoustic and textual information for real-time and accurate spoken English error correction. The model uses a transformer-based acoustic encoder to capture temporal speech features and a GCN-based module with dependency syntactic trees to model grammatical structures. A dynamic fusion mechanism effectively combines both modalities, significantly enhancing error correction. Experiments on the L2-ARCTIC and LibriSpeech corpora show our framework outperforms baseline models, achieving a 92.7% F1-score in grammatical error correction. Ablation studies confirm that incorporating grammatical information improves performance on long, complex sentences by 12.1% in F1-score. With an average response latency under 320 ms, the system meets real-time interactive requirements. This research provides valuable insights for developing robust spoken language assistance systems, with significant potential for educational and commercial applications. Inderscience Publishers - linking academia, business and industry through research

Title: Real-time error correction system of spoken English based on multimodal transformer-GCN framework

Authors: Juan Wang; Youqun Yu

Addresses: Foreign Languages Department, Southwest Jiaotong University Hope College, Chengdu, 610400, China ' Foreign Languages Department, Southwest Jiaotong University Hope College, Chengdu, 610400, China

Abstract: In today's globalised and technology-driven world, improving spoken English is increasingly important. However, traditional automatic speech recognition (ASR) systems often produce outputs with grammatical errors, poor word choices, and pronunciation ambiguities, hindering effective communication. To address this, we propose MTG-ERR, a novel multimodal transformer-GCN framework that integrates acoustic and textual information for real-time and accurate spoken English error correction. The model uses a transformer-based acoustic encoder to capture temporal speech features and a GCN-based module with dependency syntactic trees to model grammatical structures. A dynamic fusion mechanism effectively combines both modalities, significantly enhancing error correction. Experiments on the L2-ARCTIC and LibriSpeech corpora show our framework outperforms baseline models, achieving a 92.7% F1-score in grammatical error correction. Ablation studies confirm that incorporating grammatical information improves performance on long, complex sentences by 12.1% in F1-score. With an average response latency under 320 ms, the system meets real-time interactive requirements. This research provides valuable insights for developing robust spoken language assistance systems, with significant potential for educational and commercial applications.

Keywords: oral error correction; multimodal learning; transformer; graph convolutional networks; GCNs; real-time systems; grammatical dependency analysis.

DOI: 10.1504/IJICT.2026.153626

International Journal of Information and Communication Technology, 2026 Vol.27 No.50, pp.1 - 18

Received: 28 Aug 2025
Accepted: 17 Sep 2025
Published online: 18 May 2026 *

Title: Real-time error correction system of spoken English based on multimodal transformer-GCN framework

Keep up-to-date