Int. J. of Big Data Intelligence   »   2016 Vol.3, No.2

 

 

Title: A framework for collective I/O style optimisations at staging I/O nodes

 

Authors: Vishwanath Venkatesan; Mohamad Chaarawi; Quincey Koziol; Neil Fortner; Edgar Gabriel

 

Addresses:
Department of Computer Science, University of Houston, 4800 Calhoun Blvd, Houston, TX 77204, USA
The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA
The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA
The HDF Group, 1800 South Oak Street, Suite 203, Champaign, IL, 61820, USA
Department of Computer Science, University of Houston, 4800 Calhoun Blvd, Houston, TX 77204, USA

 

Abstract: Data-intensive applications are largely influenced by the I/O performance on high performance computing systems, and the scalability of such applications primarily depends on the scalability of the I/O subsystem itself. To mitigate the effects of I/O operations, recent high end systems make use of staging nodes to delegate I/O requests and thus decouple I/O and compute operations. The data staging layers however, lack the benefit that could be obtained from the collective I/O style optimisations that client side parallel I/O operations provide. In this paper, we present the compactor framework developed as a part of the exascale fastforward I/O stack. The compactor framework introduces optimisations at the data staging nodes, including collective buffering across requests from multiple processes, write stealing to service read requests at the staging node, and write morphing to optimise multiple write requests from the same process. The compactor framework is evaluated on a PVFS2 file system using micro-benchmarks, the flash I/O benchmark and a parallel image processing application. Our results indicate significant performance benefits of up to 70% due to the optimisations of the compactor.

 

Keywords: parallel I/O; input-output; data staging nodes; collective I/O; HDF5; exascale fastforward I/O; high performance computing; optimisation; parallel processing; image processing.

 

DOI: 10.1504/IJBDI.2016.077359

 

Int. J. of Big Data Intelligence, 2016 Vol.3, No.2, pp.79 - 91

 

Available online: 28 Jun 2016

 

 

Editors Full text accessAccess for SubscribersPurchase this articleComment on this article