Title: Is a picture worth a thousand words? A computational investigation of the modality effect

Authors: Naser Al Madi; Javed I. Khan

Addresses: Department of Computer Science, Kent State University, Kent, Ohio, USA ' Department of Computer Science, Kent State University, Kent, Ohio, USA

Abstract: The modality effect is a term that refers to differences in learning performance in relation to the mode of presentation. It is an interesting phenomenon that impacts education, online-learning, and marketing among many other areas of life. In this study, we use electroencephalography (EEG alpha, beta, and theta) and computational modelling of comprehension to study the modality effect in text and multimedia. First, we provide a framework for evaluating learning performance, working-memory, and emotions during learning. Second, we apply these tools to investigate the modality effect computationally focusing on text in contrast to multimedia. This study is based on a dataset that we have collected through a human experiment involving 16 participants. Our results are important for future learning systems that incorporate learning performance, working-memory, and emotions in a continuous feedback system that measures and optimises learning during learning and not after.

Keywords: modality effect; comprehension; electroencephalography; EEG; learning; education; text; multimedia; semantic networks; recall; emotions.

DOI: 10.1504/IJCSE.2019.101351

International Journal of Computational Science and Engineering, 2019 Vol.19 No.3, pp.440 - 451

Received: 11 Jul 2016
Accepted: 15 Mar 2017
Published online: 05 Aug 2019 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article