Title: On optimisation of web crawler system on Scrapy framework

Authors: Deng Kaiying; Chen Senpeng; Deng Jingwei

Addresses: School of Mathematics and Computer Science, Northwest Minzu University, Lanzhou City, Gansu Province, China ' School of Mathematics and Computer Science, Northwest Minzu University, Lanzhou City, Gansu Province, China ' School of Mathematics and Computer Science, Northwest Minzu University, Lanzhou City, Gansu Province, China

Abstract: With the continuous development of internet technology, life is accompanied by data at all times. However, network data is so complicated and confusing that it is difficult for users to find valuable information. Therefore, being able to acquire data from a vast data ocean has become an essential skill for today's business development. In this paper, a web crawler system based on the Scrapy framework is optimised to further enhance the crawler efficiency, increase the crawler speed, and break the crawler limit.

Keywords: network data; Scrapy; web crawler; optimisation.

DOI: 10.1504/IJWMC.2020.108530

International Journal of Wireless and Mobile Computing, 2020 Vol.18 No.4, pp.332 - 338

Received: 13 Oct 2018
Accepted: 08 Nov 2019
Published online: 16 Jul 2020 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article