Skip to content
Conference

Optimization of big data task scheduling algorithms in high-throughput scenarios

Nov 2026 · International Conference on Algorithms, Image Processing, and Deep Learning · Vol 14504, pp. 1450415 - 1450415-7 · 0 citations · 6 references
Engineering

Abstract

In the context of the continuous expansion of cloud computing and big data platforms, high-throughput task flows have placed higher demands on the real-time, balanced and adaptive capabilities of scheduling algorithms. This paper, targeting heterogeneous cluster environments, constructs a dynamic priority encoding mechanism that integrates task urgency, resource adaptation gain and queue pressure, designs a multi-dimensional load-aware candidate screening method, proposes a collaborative control strategy for task allocation, queue rearrangement and blocking resolution based on throughput balance, and introduces a parameter adaptive update mechanism driven by scheduling feedback. Experimental results based on 12,000 mixed tasks and an 8-node heterogeneous cluster show that the average latency of this method has dropped to 182 ms, the P95 tail latency has dropped to 326 ms, compared with FCFS, it has decreased by 42.8% and 42.0% respectively. The CPU utilization of each node remains stable between 77% and 83%, and the task completion rate in the resource jitter scenario still reaches 97.6%, indicating that this method is of great significance for improving the scheduling efficiency and operational stability of big data platforms in high-throughput scenarios.

View source

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.