Title: Data quality improvement in data warehouse: a framework

Authors: Rajiv Arora; Payal Pahwa; Daya Gupta

Addresses: Computer Engineering Department, Delhi Technological University, New Delhi, India ' Computer Engineering Department, BPIT, Guru Gobind Singh Indraprastha University, New Delhi, India ' Computer Engineering Department, Delhi Technological University, New Delhi, India

Abstract: Data cleansing is an extremely imperative process which when carried out on the datasets, eliminates the inconsistency and duplicity from the data. It also handles null values or missing values in the data in an organised and proper manner thereby enhancing the quality of the data. In this paper, we use Kullback-Leibler divergence (KL-divergence) technique to eliminate duplicity in the datasets. Inconsistency, null values or missing values are also handled in the datasets. This is done by maintaining data marts which are made on the basis of test data. Accordingly, a framework for efficient data cleansing is suggested in order to make the data appropriate and proper for decision making purpose. A brief comparison of existing approaches of data cleansing have also been discussed. This comparison is based on various parameters such as prediction error, bias, mean square error, variance, mean absolute error, root mean square error, Theil statistics etc. These parameters are used by distance sum-based approach (DSA) to accomplish the task. The results obtained demonstrate the feasibility and validity of our method.

Keywords: Kullback-Leibler divergence; KL-divergence; data cleansing; data pruning; distance sum-based approach; DSA; data quality improvement; data warehouse; prediction error; bias; mean square error; variance; mean absolute error; root mean square error; RMSE; Theil statistics; data warehousing.

DOI: 10.1504/IJDATS.2017.083062

International Journal of Data Analysis Techniques and Strategies, 2017 Vol.9 No.1, pp.17 - 33

Published online: 20 Mar 2017 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article