Title: Virtual cluster optimisation for MapReduce-like applications

Authors: Cairong Yan; Guangwei Xu

Addresses: School of Computer Science and Technology, Donghua University, Shanghai, China ' School of Computer Science and Technology, Donghua University, Shanghai, China

Abstract: Infrastructure-as-a-service clouds are becoming ubiquitous for provisioning virtual machines on demand. Cloud service providers expect to use the least resources to deliver the best services. As users frequently request virtual machines to build virtual clusters and run MapReduce-like jobs for big data processing, cloud service providers intend to optimise the virtual cluster to minimise network latency and subsequently reduce data movement cost. In this paper, we focus on the virtual machine placement issue for provisioning virtual clusters with minimum network latency in clouds. We define the distance as the latency between virtual machines and use it to measure the affinity of a virtual cluster. Such metric of distance indicates the considerations of virtual machine placement and the topology of physical nodes in clouds. Then, we formulate our problem as the classical shortest distance problem and solve it by building an integer programming model. A greedy virtual machine placement algorithm is designed to get a compact virtual cluster. Furthermore, an improved heuristic algorithm is also presented for achieving a global resource optimisation. The simulation results verify our algorithms and the experiment results validate the improvement achieved by our approaches.

Keywords: virtual cluster; provisioning; resource optimisation; MapReduce programming model; shortest distance.

DOI: 10.1504/IJHPCN.2019.099262

International Journal of High Performance Computing and Networking, 2019 Vol.13 No.4, pp.378 - 388

Received: 09 May 2016
Accepted: 17 Oct 2016
Published online: 24 Apr 2019 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article