Title: Benchmarking short text semantic similarity

Authors: James O'Shea, Zuhair Bandar, Keeley Crockett, David McLean

Addresses: Department of Computing and Mathematics, Manchester Metropolitan University, John Dalton Building, Chester St., Manchester M1 5GD, UK. ' Department of Computing and Mathematics, Manchester Metropolitan University, John Dalton Building, Chester St., Manchester M1 5GD, UK. ' Department of Computing and Mathematics, Manchester Metropolitan University, John Dalton Building, Chester St., Manchester M1 5GD, UK. ' Department of Computing and Mathematics, Manchester Metropolitan University, John Dalton Building, Chester St., Manchester M1 5GD, UK

Abstract: Short text semantic similarity measurement is a new and rapidly growing field of research. |Short texts| are typically sentence length but are not required to be grammatically correct. There is great potential for applying these measures in fields such as information retrieval, dialogue management and question answering. A dataset of 65 sentence pairs, with similarity ratings, produced in 2006 has become adopted as a de facto gold standard benchmark. This paper discusses the adoption of the 2006 dataset, lays down a number of criteria that can be used to determine whether a dataset should be awarded a |gold standard| accolade and illustrates its use as a benchmark. Procedures for the generation of further gold standard datasets in this field are recommended.

Keywords: short texts; short text similarity; sentence similarity; semantic similarity; benchmarking; datasets; gold standard; information retrieval; dialogue management; question answering; intelligent information systems; intelligent database systems; semantic distance; semantics.

DOI: 10.1504/IJIIDS.2010.032437

International Journal of Intelligent Information and Database Systems, 2010 Vol.4 No.2, pp.103 - 120

Accepted: 27 Nov 2009
Published online: 02 Apr 2010 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article