Title: Using latent semantic analysis to grade brief summaries: some proposals

Authors: Ricardo Olmos, Jose A. Leon, Inmaculada Escudero, Guillermo Jorge-Botana

Addresses: Departamento de Psicologia Basica, Facultad de Psicologia, Universidad Autonoma de Madrid, Campus de Cantoblanco, 28049 Madrid, Spain. ' Departamento de Psicologia Basica, Facultad de Psicologia, Universidad Autonoma de Madrid, Campus de Cantoblanco, 28049 Madrid, Spain. ' Departamento de Psicologia Evolutiva, Facultad de Psicologia, UNED, 28080 Madrid, Spain. ' Departamento de Psicologia Basica, Facultad de Psicologia, Universidad Autonoma de Madrid, Campus de Cantoblanco, 28049 Madrid, Spain

Abstract: In this paper, we present several proposals in order to improve the LSA tools to evaluate brief summaries (less than 50 words) of narrative and expository texts. First, we analyse the quality of six different methods assessing essays that have been widely employed before (Foltz et al., 2000). The second objective is to analyse how new algorithms inspired by some authors (Denhiere et al., 2007) that try to emulate human behaviour to improve the reliability of LSA with human graders when assessing short summaries, compared with standard LSA use in expository text. Finally, we present an assessment method to combine LSA as a semantic computational linguistic model with ROUGE-N as a lexical model, to show how combining different automatic evaluation systems (LSA and ROUGE) can improve the quality of assessments in different academic levels.

Keywords: latent semantic analysis; LSA; assessment summaries; ROUGE-N; automated scoring; brief summaries; vector length; expository texts; narrative texts; grades; grading; essays; algorithms; human behaviour; behaviour emulation; reliability; human graders; short summaries; computational linguistics; lexical models; automatic evaluation systems; academic levels; assessment quality; Peter Foltz; Spain; Guy Denhiere; continuing education; life-long learning; automatic free-text evaluation.

DOI: 10.1504/IJCEELL.2011.040198

International Journal of Continuing Engineering Education and Life-Long Learning, 2011 Vol.21 No.2/3, pp.192 - 209

Available online: 20 May 2011 *

Full-text access for editors Access for subscribers Purchase this article Comment on this article