Title: Optimising job scheduling in Apache Spark: a simulation framework for algorithm comparison and performance prediction

Authors: Vishnu Prasad Verma; Santosh Kumar; Nenavath Srinivas Naik

Addresses: Department of Computer Science Engineering, International Institute of Information Technology, Naya Raipur, Raipur, 493661, Chhattisgarh, India ' Department of Computer Science Engineering, International Institute of Information Technology, Naya Raipur, Raipur, 493661, Chhattisgarh, India ' Department of Computer Science and Engineering, Indian Institute of Information Technology, Design and Manufacturing, Kurnool, Kurnool, 518008, Andhra Pradesh, India

Abstract: Selecting the best scheduling algorithm is essential for optimising Apache Spark's performance and resource usage. This study conducts a comparative analysis of different scheduling algorithms, including first in, first out (FIFO), fair, earliest deadline first (EDF), round robin, shortest job next, least laxity first, priority scheduler, and multilevel feedback queue using a Python-based simulation framework. The analysis focuses on critical performance metrics such as turnaround time, waiting time, deadline adherence, and violation rates. Our findings highlight notable performance differences among scheduling algorithms when tested under various workload conditions. Among them, the multilevel feedback queue (MLFQ) consistently emerged as the top performer, achieving the lowest turnaround and waiting times, a remarkable earliest arrival ratio of 97.3%, and no deadline violations. The experimental findings guide researchers, cloud infrastructure designers, and Spark system architects in successfully choosing and refining scheduling techniques to manage various dynamic workload needs.

Keywords: distributed computing; Spark job scheduling; big data; scheduling algorithms; resource management; optimisation.

DOI: 10.1504/IJAHUC.2026.155464

International Journal of Ad Hoc and Ubiquitous Computing, 2026 Vol.52 No.3, pp.145 - 157

Received: 04 Nov 2024
Accepted: 16 Jun 2025
Published online: 03 Aug 2026 *

Full-text access for editors Full-text access for subscribers Purchase this article Comment on this article