Title: Empirical software engineering for evaluating adaptive task-oriented personal assistants: a case study in human/machine event extraction and coding

Authors: Wayne Wobcke; Alfred Krzywicki

Addresses: School of Computer Science and Engineering, University of New South Wales, Sydney NSW 2052, Australia; UNSW Data Science Hub (uDASH), University of New South Wales, Sydney NSW 2052, Australia ' School of Computer Science and Engineering, University of New South Wales, Sydney NSW 2052, Australia

Abstract: This paper concerns methodology for evaluating task-oriented personal assistants, where users perform a complex task that has objective success criteria, independent of personal preferences, and the agent provides suggestions to help users repeatedly perform the task consistently and accurately. We develop a systematic approach to evaluating task-oriented personal assistants in normal contexts of use through extending the methodology of empirical software engineering to evaluate effectiveness, efficiency and satisfaction. The approach allows the evaluation of both the human-agent system and of the personal assistant using data obtained by observations of user and system behaviour. A key element of our approach is to define empirically observable conditions that separate the learning period, when users and the agent are learning to perform the task, from the evaluation period, when performance benefits are measured. The methodology is illustrated using the example of a system for users to extract, annotate and code events from news articles.

Keywords: personal agent; empirical software engineering; event extraction.

DOI: 10.1504/IJAOSE.2022.122647

International Journal of Agent-Oriented Software Engineering, 2022 Vol.7 No.2, pp.184 - 216

Received: 10 Jun 2021
Accepted: 09 Jan 2022
Published online: 04 May 2022 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article