Title: Assessment of web article's completeness by capturing discussed topics

Authors: Jingyu Han; Kejia Chen; Zhu Zheng; Yanying Yang

Addresses: College of Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China ' College of Computer Science and Technology, Nanjing University of Posts and Telecommunications, Nanjing, 210003, China ' Department of Computer Science and Information Systems, Birkbeck College, University of London, London WC1E 7HX, UK ' Department of Information and Technology, Nanjing College of Forest Police, Nanjing, 210023, China

Abstract: Completeness is one of the first-class data quality dimensions of web articles. From the point of view of the semantics, we propose to measure completeness in terms of topic-coverage and topic-depth by capturing an article's relation distribution. Given a target article, we use a generative probabilistic model, latent Dirichlet allocation (LDA), to generate its topic-coverage and topic-depth baselines by leveraging knowledge from Wikipedia. It consists of two phases, the baseline construction and the completeness calculation. During the baseline construction, we extract Wikipedia knowledge to construct baselines using the LDA generative process, which present what should be covered and how deeply each sub-topic should be discussed. During the completeness calculation, the completeness is quantified by comparing the article with its topic-coverage and topic-depth baselines. In the meantime, what should be discussed by the article is also predicted. Experiments demonstrate that our approach can effectively rate completeness and predict content.

Keywords: data quality; web articles; article completeness; latent Dirichlet allocation; LDA; semantics; topic coverage; topic depth; relations distribution; generative probabilistic modelling; content prediction.

DOI: 10.1504/IJWET.2014.065871

International Journal of Web Engineering and Technology, 2014 Vol.9 No.3, pp.247 - 276

Published online: 17 Dec 2014 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article