Title: Studies on intrinsic summary evaluation

Authors: Shanmugasundaram Hariharan, Rengaramanujam Srinivasan

Addresses: Department of Information Technology, School of Computer and Information Sciences, B.S. Abdur Rahman University, Chennai 600048, Tamilnadu, India. ' Department of Computer Science and Engineering, School of Computer and Information Sciences, B.S. Abdur Rahman University, Chennai 600048, Tamilnadu, India

Abstract: Summary evaluation has gained importance in the research community and seems to be a quite challenging task. Most of the work till date focus on summary generation, while this paper focus on intrinsic summary evaluation. Studies made for preparing the |Gold Standard| reference summary present some of the key issues in preparing a reference summary. Spearman, Kendall|s W, and Kendall|s rank correlation coefficients have been used for measuring inter-judge rank correlation. Existing drawbacks in Precision-Recall based evaluation and utility-based evaluation schemes were analysed. A new measure, called |Effectiveness|, to measure the system performance at given compression rate has been introduced.

Keywords: rank correlation; inter judge agreement; intrinsic summary evaluation; extraction; summarisation; utility mechanisms; precision; recall; text summary; effectiveness.

DOI: 10.1504/IJAISC.2010.032513

International Journal of Artificial Intelligence and Soft Computing, 2010 Vol.2 No.1/2, pp.58 - 76

Published online: 04 Apr 2010 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article