Title: Topical document clustering: two-stage post processing technique

Authors: Poonam Goyal; N. Mehala; Divyansh Bhatia; Navneet Goyal

Addresses: Department of Computer Science, Birla Institute of Technology and Science, 333 031, Pilani, India ' Department of Computer Science, Presidency University, 560089, Bengaluru, India ' eBay Inc., San Jose, USA ' Department of Computer Science, Birla Institute of Technology and Science, 333 031, Pilani, India

Abstract: Clustering documents is an essential step in improving efficiency and effectiveness of information retrieval systems. We propose a two-phase split-merge (SM) algorithm, which can be applied to topical clusters obtained from existing query-context-aware document clustering algorithms, to produce soft topical document clusters. The SM is a post-processing technique which combines the advantages of document and feature-pivot topical document clustering approaches. The split phase splits the topical clusters by relating them to the topics obtained by disambiguating web search results, and converts them into homogeneous soft clusters. In the merge phase, similar clusters are merged by feature-pivot approach. The SM is tested on the outcome of two hierarchical query-context aware document clustering algorithms on different datasets including TREC session-track 2011 dataset. The obtained topical-clusters are also updated by an incremental approach with the progress in the data stream. The proposed algorithm improves the quality of clustering appreciably in all the experiments conducted.

Keywords: topical clustering; query clustering; query context; document clustering; incremental clustering; soft clustering.

DOI: 10.1504/IJDMMM.2018.092536

International Journal of Data Mining, Modelling and Management, 2018 Vol.10 No.2, pp.127 - 170

Received: 01 Sep 2016
Accepted: 08 Sep 2017

Published online: 24 Jun 2018 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article