Title: Evaluating classification accuracy: the impact of resampling and dataset size

Authors: Jehad Imlawi; Mohammad Alsharo

Addresses: Information Systems Department, Al al-Bayt University, P.O. Box 130040, Mafraq, 25113, Jordan ' Information Systems Department, Al al-Bayt University, P.O. Box 130040, Mafraq, 25113, Jordan

Abstract: Correct prediction is important criterion in evaluating classifiers in supervised learning context. The accuracy rate is a widely accepted indicator of the probability of misclassification of a classifier. Nevertheless, true accuracy remains unknown in most cases since it is not always possible to include the whole population in a study, and it is difficult to calculate the probability distribution of the data. Therefore, researchers often rely on computing estimation from the available data through sampling. When the available data is small or limited, it is common to rely on a resampling technique for accuracy estimation. In this paper, we study the impact of the resampling against non-resampling estimation method, with different dataset sizes on the sample distribution variance. Initial results indicate that there is a significant difference in the variance of the sample distribution between resampling and non-resampling. We also found that the larger the dataset size, the less significant the difference in variance.

Keywords: resampling; dataset size; classification accuracy; cross validation; distribution variance; classifier evaluation; supervised learning.

DOI: 10.1504/IJBIS.2017.080947

International Journal of Business Information Systems, 2017 Vol.24 No.1, pp.91 - 101

Received: 11 May 2015
Accepted: 28 Jun 2015
Published online: 13 Dec 2016 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article