Article: Semantic association and cognitive-level adaptation of multimodal teaching resources using knowledge forest and transformer models Journal: International Journal of Information and Communication Technology (IJICT) 2026 Vol.27 No.86 pp.87 - 109 Abstract: This study addresses the mismatch between proliferating multimodal resources and learners' cognitive needs in digital education. Conventional manual annotation and static classification fail to capture cross-modal semantic associations or adapt to evolving cognitive levels. We propose a knowledge forest-transformer model integrating hierarchical knowledge organisation with self-attention mechanisms for multimodal semantic association and cognitive adaptation. Structured multimodal feature modelling improves cross-modal accuracy: text-image from 68.2% to 89.7%, text-audio from 59.8% to 83.6%, and image-video to 90.2%. Cognitive adaptation achieves 85.8% success at threshold 1.8, while cognitive labels enhance F1-scores (analysis layer: 61.2% to 79.5%; application layer: +17.8%). Response times under three seconds yield cognitive load of 4.2 (12.5% exceedance) versus 8.4 (68% exceedance) when exceeding eight seconds. After four weeks, low-ability learners improve mastery from 32.1% to 61.3%, with medium and high-ability groups reaching 82.7% and 94.2%, respectively. Results demonstrate effective multimodal resource association and cognitive adaptive support. Inderscience Publishers - linking academia, business and industry through research

Title: Semantic association and cognitive-level adaptation of multimodal teaching resources using knowledge forest and transformer models

Authors: Shiping Liu

Addresses: School of Educational Sciences, Henan Finance University, Zhengzhou, 450046, China

Abstract: This study addresses the mismatch between proliferating multimodal resources and learners' cognitive needs in digital education. Conventional manual annotation and static classification fail to capture cross-modal semantic associations or adapt to evolving cognitive levels. We propose a knowledge forest-transformer model integrating hierarchical knowledge organisation with self-attention mechanisms for multimodal semantic association and cognitive adaptation. Structured multimodal feature modelling improves cross-modal accuracy: text-image from 68.2% to 89.7%, text-audio from 59.8% to 83.6%, and image-video to 90.2%. Cognitive adaptation achieves 85.8% success at threshold 1.8, while cognitive labels enhance F1-scores (analysis layer: 61.2% to 79.5%; application layer: +17.8%). Response times under three seconds yield cognitive load of 4.2 (12.5% exceedance) versus 8.4 (68% exceedance) when exceeding eight seconds. After four weeks, low-ability learners improve mastery from 32.1% to 61.3%, with medium and high-ability groups reaching 82.7% and 94.2%, respectively. Results demonstrate effective multimodal resource association and cognitive adaptive support.

Keywords: multimodal teaching resources; semantic association; cognitive level adaptation; knowledge forest; transformer.

DOI: 10.1504/IJICT.2026.155559

International Journal of Information and Communication Technology, 2026 Vol.27 No.86, pp.87 - 109

Received: 26 Feb 2026
Accepted: 16 Apr 2026
Published online: 05 Aug 2026 *

Title: Semantic association and cognitive-level adaptation of multimodal teaching resources using knowledge forest and transformer models

Keep up-to-date