Article: Design and implementation of an experimental big data management platform architecture integrating LSH and columnar storage optimisation Journal: International Journal of Computational Science and Engineering (IJCSE) 2026 Vol.29 No.7 pp.1 - 11 Abstract: To resolve the mismatch between the local I/O advantages of columnar storage and the global vector operations required by locality-sensitive hashing (LSH), this paper designs and implements an experimental big data management platform integrating LSH with columnar storage optimisation. The architecture employs a four-layer experimental awareness framework incorporating dynamic column grouping, bucket-aware index reconstruction, and collaborative query optimisation to enhance storage efficiency and retrieval performance. This integration bridges the gap between analytical engines and indexing mechanisms by coupling LSH logic directly with the physical storage tier. Performance evaluations on a 100-node cluster reveal that the system achieves an I/O overhead of 22.0 GB and a P99 query latency of 102 ms for 10,000-dimensional data. This platform provides a robust and scalable infrastructure for managing large-scale high-dimensional experimental data, offering significant implications for reproducible scientific research and high-performance data analytics. Inderscience Publishers - linking academia, business and industry through research

Title: Design and implementation of an experimental big data management platform architecture integrating LSH and columnar storage optimisation

Authors: Jun Li

Addresses: School of Information Technology Engineering, Taizhou Vocational & Technical College, Taizhou, Zhejiang 318000, China

Abstract: To resolve the mismatch between the local I/O advantages of columnar storage and the global vector operations required by locality-sensitive hashing (LSH), this paper designs and implements an experimental big data management platform integrating LSH with columnar storage optimisation. The architecture employs a four-layer experimental awareness framework incorporating dynamic column grouping, bucket-aware index reconstruction, and collaborative query optimisation to enhance storage efficiency and retrieval performance. This integration bridges the gap between analytical engines and indexing mechanisms by coupling LSH logic directly with the physical storage tier. Performance evaluations on a 100-node cluster reveal that the system achieves an I/O overhead of 22.0 GB and a P99 query latency of 102 ms for 10,000-dimensional data. This platform provides a robust and scalable infrastructure for managing large-scale high-dimensional experimental data, offering significant implications for reproducible scientific research and high-performance data analytics.

Keywords: big data management; columnar storage; locality sensitive hashing; LSH; experimental platform.

DOI: 10.1504/IJCSE.2026.156819

International Journal of Computational Science and Engineering, 2026 Vol.29 No.7, pp.1 - 11

Received: 17 Dec 2025
Accepted: 26 Jul 2026
Published online: 06 Oct 2026 *

Title: Design and implementation of an experimental big data management platform architecture integrating LSH and columnar storage optimisation

Keep up-to-date