Title: Architecture specification of rule-based deep web crawler with indexer

Authors: S.G. Shaila; A. Vadivel

Addresses: Department of Computer Applications, Multimedia Information Retrieval Group, National Institute of Technology, Tiruchirappalli 620 015, India ' Department of Computer Applications, Multimedia Information Retrieval Group, National Institute of Technology, Tiruchirappalli 620 015, India

Abstract: Suitable architecture specification of a deep web crawler with surface web crawler as well as indexer is proposed for fetching large number of documents from deep web using rules. The functional dependency of core and allied fields in the FORM are identified for generating rules using SVM classifier and classifies them as most preferable, least preferable and mutually exclusive. The FORMs are filled with values from most preferable class for fetching large number of documents. The extracted document is indexed for information retrieval applications. The architecture is extended to distributed crawler using web services. The proposed crawler fetches large number of documents while using the values in most preferable class. This architecture has higher coverage rate and reduces fetching time. The retrieval performance is encouraging and achieves similar precision of retrieval as Google search engine system.

Keywords: information retrieval; surface web; deep web crawlers; rules; web services; indexers; distributed crawlers; SVM; support vector machines.

DOI: 10.1504/IJKWI.2013.056366

International Journal of Knowledge and Web Intelligence, 2013 Vol.4 No.2/3, pp.166 - 186

Received: 27 Sep 2012
Accepted: 22 Dec 2012

Published online: 26 Jul 2014 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article