Title: Research on machine reading comprehension for BERT and its variant neural network models

Authors: YanFeng Wang; Ning Ma; Wenrong Lv

Addresses: Gansu Key Laboratory of National Language Intelligent Processing, Northwest Minzu University, Lanzhou, Gansu, China; School of Mathematics and Computer Science, Northwest Minzu University, Lanzhou, Gansu, China ' Key Laboratory of Chinese National Language and Character Information Technology, Northwest Minzu University, Lanzhou, Gansu, China; Gansu Key Laboratory of National Language Intelligent Processing, Northwest Minzu University, Lanzhou, Gansu, China ' Key Laboratory of Chinese National Language and Character Information Technology, Northwest Minzu University, Lanzhou, Gansu, China; Gansu Key Laboratory of National Language Intelligent Processing, Northwest Minzu University, Lanzhou, Gansu, China

Abstract: Currently, machine reading comprehension models primarily rely on the LSTM networks with the gate mechanism. In the present study, we employ BERT and its variant pre-training-trained language model to conduct research and experimentation on the DuReader data set. We find that enhanced masking techniques, such as full-word mask masking and dynamic mask masking, can significantly enhance the model's performance in machine reading comprehension tasks. Therefore, the ROUGE-L and BLEU-4 values of RoBERTRoBERTa-wwm-ext model on the test set achieve ROUGE-L and BLEU-4 scores of 51.02% and 48.14%, respectively, which are 19.12% and 8.94% higher than the benchmark model. In addition, addressing the issue of suboptimal model performance is not optimal when the data is dealing with large data and the effective information is relatively dispersed. This paper employs a three-step pre-processing approach for the data set. This method is based on the F1-score to identify relevant paragraphs, answer modules and feature precomputation, so that the performance of precompute features, ultimately bringing the pre-training-trained language model, is shown to bring the model's performance closer to the average human reading comprehension level.

Keywords: machine reading comprehension; BERT pre-training language model; masking mode.

DOI: 10.1504/IJCAT.2025.148136

International Journal of Computer Applications in Technology, 2025 Vol.76 No.1/2, pp.27 - 35

Received: 25 May 2023
Accepted: 18 Jan 2024
Published online: 27 Aug 2025 *