Title: Determining the level of agreement of expert estimates

Authors: Valentinas Podvezko

Addresses: Department of Mathematical Statistics, Vilnius Gediminas Technical University, Sauletekio al. 11, Vilnius-40 LT-10223, Lithuania

Abstract: This paper considers the techniques of determining the concordance of expert estimates. Expert methods of evaluating particular criteria may be practically used, for instance, in multi-criteria decision support systems, if judgements are in good agreement. This is expressed by the concordance coefficient calculated by using expert-provided data. The concordance coefficient can be calculated only if the considered criteria are ranked. If other evaluation methods are used by experts, the data should be transformed to rankings before the calculation of the concordance coefficient is made. There is the dependence of the agreement of expert estimates on a particular method used. The highest degree of agreement has been obtained by using a direct ranking method. The effect of equally assessed criteria, that is, the tied ranks, on the concordance coefficient and thereby on the level of expert judgements agreement is usually insignificant and cannot change the results of rating.

Keywords: expert estimates; judgement agreement; ranking; concordance coefficient; data normalisation; criteria evaluation; multicriteria decision making; MCDM; decision support systems; DSS; expert judgements.

DOI: 10.1504/IJMDM.2007.013420

International Journal of Management and Decision Making, 2007 Vol.8 No.5/6, pp.586 - 600

Published online: 27 Apr 2007 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article