Title: Development of intelligent data collection and management system based on internet of things big data crawler technology

Authors: Jin Chen; Yao Li; Xia Hua; Long Lu

Addresses: School of Economics and Trade, Haojing College of Shaanxi University of Science and Technology, Xianyang, Shaanxi, China ' School of Computer Science, Haojing College of Shaanxi University of Science and Technology, Xianyang, Shaanxi, China ' School of Economics and Trade, Haojing College of Shaanxi University of Science and Technology, Xianyang, Shaanxi, China ' School of Economics and Trade, Haojing College of Shaanxi University of Science and Technology, Xianyang, Shaanxi, China

Abstract: This paper used web crawler technology to develop an intelligent data collection and management system. This paper first analysed the basic principles of system design and the structural requirements of the system according to the system requirements. Then, it evaluated the overall results of the intelligent data collection and management system. The process and principle of information collection using the IoT Big Data (BD) crawler technology were introduced in detail. Finally, the information collection effect of the system was verified by experiments. The experimental results showed that the data acquisition accuracy of the system was high, accounting for more than 90%. The system had high usability and efficiency; users were satisfied, and data collection and management could be done well. Existing research is often limited by the dispersion and heterogeneity of data sources when collecting data. Data in the IoT environment usually has problems with diverse formats and complex protocols.

Keywords: intelligent data collection; data management; big data of the internet of things; big data crawler technology.

DOI: 10.1504/IJCAT.2025.149847

International Journal of Computer Applications in Technology, 2025 Vol.77 No.1/2, pp.11 - 22

Received: 19 Oct 2024
Accepted: 02 Jan 2025
Published online: 14 Nov 2025 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article