Title: Architecture specification of rule-based deep web crawler with indexer
Authors: S.G. Shaila; A. Vadivel
Addresses: Department of Computer Applications, Multimedia Information Retrieval Group, National Institute of Technology, Tiruchirappalli 620 015, India ' Department of Computer Applications, Multimedia Information Retrieval Group, National Institute of Technology, Tiruchirappalli 620 015, India
Abstract: Suitable architecture specification of a deep web crawler with surface web crawler as well as indexer is proposed for fetching large number of documents from deep web using rules. The functional dependency of core and allied fields in the FORM are identified for generating rules using SVM classifier and classifies them as most preferable, least preferable and mutually exclusive. The FORMs are filled with values from most preferable class for fetching large number of documents. The extracted document is indexed for information retrieval applications. The architecture is extended to distributed crawler using web services. The proposed crawler fetches large number of documents while using the values in most preferable class. This architecture has higher coverage rate and reduces fetching time. The retrieval performance is encouraging and achieves similar precision of retrieval as Google search engine system.
Keywords: information retrieval; surface web; deep web crawlers; rules; web services; indexers; distributed crawlers; SVM; support vector machines.
DOI: 10.1504/IJKWI.2013.056366
International Journal of Knowledge and Web Intelligence, 2013 Vol.4 No.2/3, pp.166 - 186
Received: 27 Sep 2012
Accepted: 22 Dec 2012
Published online: 26 Jul 2014 *