Title: The evaluation of adaptive and personalised information retrieval systems: a review

Authors: Catherine Mulwa; Seamus Lawless; Mary Sharp; Vincent Wade

Addresses: Centre for Next Generation Localisation, Knowledge and Data Engineering Group, School of Computer Science and Statistics, Trinity College, Dublin, Ireland ' Centre for Next Generation Localisation, Knowledge and Data Engineering Group, School of Computer Science and Statistics, Trinity College, Dublin, Ireland ' Centre for Next Generation Localisation, Knowledge and Data Engineering Group, School of Computer Science and Statistics, Trinity College, Dublin, Ireland ' Centre for Next Generation Localisation, Knowledge and Data Engineering Group, School of Computer Science and Statistics, Trinity College, Dublin, Ireland

Abstract: A current problem with the research of adaptive systems is the inconsistency of evaluation applied to the adaptive systems. However, evaluating an adaptive system is a difficult task due to the complexity of such systems. Evaluators need to ensure correct evaluation methods and measurement metrics are used. This paper reviews a variety of evaluation techniques applied in adaptive and user-adaptive systems. More specifically, it focuses on the user-centred evaluation of adaptive systems such as personalised recommender systems and adaptive information retrieval systems. The review tackles the question of ‘How have user-centred evaluations of adaptive and user-adaptive systems been conducted and how can these evaluation practices be improved?’ Based on the analysed results of the: (a) evaluation approaches, (b) user-centred evaluation techniques, and (c) evaluation metrics, we propose an evaluation framework for end-user experience in evaluating adaptive systems (EFEx).

Keywords: adaptive information retrieval systems; user-adaptive systems; user-centred evaluations; personalised information retrieval systems.

DOI: 10.1504/IJKWI.2011.044120

International Journal of Knowledge and Web Intelligence, 2011 Vol.2 No.2/3, pp.138 - 156

Available online: 09 Dec 2011 *

Full-text access for editors Access for subscribers Purchase this article Comment on this article