Title: A framework for utilising usage trends in the crawling and indexing process of search engines

Authors: Neelam Duhan; A.K. Sharma

Addresses: Department of Computer Engineering, YMCA University of Science and Technology, Zakir Nagar, Sector-6, Faridabad, India. ' Department of Computer Engineering, YMCA University of Science and Technology, Zakir Nagar, Sector-6, Faridabad, India

Abstract: Making search engines responsive to human needs requires understanding of user navigations through the search results in response to the submitted queries. The user behaviour characterisation provides an interesting perspective towards understanding the workload imposed on the search engine and can be used to address crucial points such as load balancing, content caching, data distribution and result optimisation. The user browsing behaviour is recorded in the query logs of search engines and usually referred to as web usage data. In this paper, a technique to utilise the users' browsing behaviour at the crawling and indexing process is being proposed so as to direct the crawler to download the important pages, which were not previously crawled. As the work attempts to index most of important pages based on user feedback, it would benefit the search engine to enhance its efficiency. To add further to the proposed work, the existing data structures maintained by the search engines has been refined so as to support the proposed user feedback mechanism and open more research directions.

Keywords: World Wide Web; internet; search engines; crawlers; indexing; query logs; usage trends; user behaviour; crawling; load balancing; content caching; data distribution; results optimisation; browsing behaviour; search engine efficiency.

DOI: 10.1504/IJKWI.2011.045164

International Journal of Knowledge and Web Intelligence, 2011 Vol.2 No.4, pp.272 - 291

Published online: 07 Mar 2015 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article