Authors: Qingxuan Meng; Jianzhuo Yan
Addresses: College of Electronic Information and Control Engineering, Beijing University of Technology, Beijing, China ' College of Electronic Information and Control Engineering, Beijing University of Technology, Beijing, China
Abstract: Identifying and rectifying incomplete water quality data is of vital importance. A data cleaning method based on improved balanced iterative reducing and clustering using hierarchies (BIRCH) clustering algorithm is proposed. The clustering feature tree of water quality data is constructed and the cluster vector of the clustering feature tree is obtained by the agglomerative method. The optimal cluster number is determined according to the Bayesian Information Criterion and the nearest clustering ratio. The Pauta criterion is used to detect the global outlier and artificial neural network (ANN) is used to fill in outliers and missing values. Finally, the improved data cleaning method is applied to water quality monitoring data of Beijing wastewater treatment plant. The experimental results show that the data cleaning method can not only detect abnormal values and missing values accurately, but also normalise and complete missing data.
Keywords: outliers; water quality monitoring; multivariate data; clustering; artificial neural network; ANN.
International Journal of Simulation and Process Modelling, 2019 Vol.14 No.5, pp.442 - 451
Received: 04 Sep 2018
Accepted: 04 Mar 2019
Published online: 09 Dec 2019 *