Title: Designing ensemble learning algorithms using kernel methods

Authors: Fayao Liu; Ruizhi Qiao; Chunhua Shen; Lei Luo

Addresses: School of Computer Science, University of Adelaide, Australia ' School of Computer Science, University of Adelaide, Australia ' School of Computer Science, University of Adelaide, Australia ' College of Computer, National University of Defence Technology, China

Abstract: Ensemble methods such as boosting combine multiple learners to obtain better prediction than could be obtained from any individual learner. Here we propose a principled framework for directly constructing ensemble learning methods from kernel methods. Unlike previous studies showing the equivalence between boosting and support vector machines (SVMs) which need a translation procedure, we show that it is possible to design boosting-like procedure to solve the SVM optimisation problems. In other words, it is possible to design ensemble methods directly from SVM without any middle procedure. This finding not only enables us to design new ensemble learning methods directly from kernel methods, but also makes it possible to take advantage of those highly-optimised fast linear SVM solvers for ensemble learning. The resulted model is as effective as kernel methods while being as efficient as ensemble methods. We exemplify this framework for designing new binary and multi-class classification ensemble learning as well as a new quantile regression ensemble learning method. Experimental results demonstrate the flexibility and usefulness of the proposed framework.

Keywords: kernel; support vector machines; SVMs; ensemble learning; column generation; multi-class classification; quantile regression.

DOI: 10.1504/IJMISSP.2017.088165

International Journal of Machine Intelligence and Sensory Signal Processing, 2017 Vol.2 No.1, pp.1 - 31

Received: 09 Jan 2017
Accepted: 10 Jan 2017
Published online: 27 Nov 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article