Title: Understanding expert ratings of essay quality: Coh-Metrix analyses of first and second language writing

Authors: Scott A. Crossley, Danielle S. McNamara

Addresses: Department of Applied Linguistics, Georgia State University, 34 Peachtree St. Suite 1200, One Park Tower Building, Atlanta, GA 30303, USA. ' Psychology/Institute for Intelligent Systems, University of Memphis, Memphis, TN 38152, USA

Abstract: This article reviews recent studies in which human judgements of essay quality are assessed using Coh-Metrix, an automated text analysis tool. The goal of these studies is to better understand the relationship between linguistic features of essays and human judgements of writing quality. Coh-Metrix reports on a wide range of linguistic features, affording analyses of writing at various levels of text structure, including surface, text-base, and situation model levels. Recent studies have examined linguistic features of essay quality related to co-reference, connectives, syntactic complexity, lexical diversity, spatiality, temporality, and lexical characteristics. These studies have analysed essays written by both first language and second language writers. The results support the notion that human judgements of essay quality are best predicted by linguistic indices that correlate with measures of language sophistication such as lexical diversity, word frequency, and syntactic complexity. In contrast, human judgements of essay quality are not strongly predicted by linguistic indices related to cohesion. Overall, the studies portray high quality writing as containing more complex language that may not facilitate text comprehension.

Keywords: writing assessments; cohesion; essay quality; lexical development; syntactic development; computational linguistics; expert ratings; Coh-Metrix analyses; first language; second language; human judgements; automated text analysis; linguistic features; essays; text structures; surface levels; text-base levels; situation model levels; co-reference; connectives; syntactic complexity; lexical diversity; spatiality; temporality; lexical characteristics; linguistic indices; language sophistication; word frequency; complex language; text comprehension; continuing education; life-long learning; automatic free-text evaluation.

DOI: 10.1504/IJCEELL.2011.040197

International Journal of Continuing Engineering Education and Life-Long Learning, 2011 Vol.21 No.2/3, pp.170 - 191

Published online: 20 May 2011 *

Full-text access for editors Access for subscribers Purchase this article Comment on this article