An Efficient Computing and Communication Framework for Large-Scale Data Processing Cluster

Abstract

Data processing clusters dealing with big data are facing extended completion times for jobs because the RDMA feature is not being used efficiently. Our production data from a large cluster, which has many server nodes and is responsible for processing extensive data jobs, reveals that the current use of the RDMA technology is causing some jobs to finish much later than usual, with a few taking more than double the standard time to be completed. In this paper, we introduce the design and implementation of Turbo, a high-performance, scalable communication framework tailored for large-scale data processing clusters. The essence of Turbo’s strategy lies in the use of a dynamic block-level flowlet transmission system and a non-blocking communication middleware, which are designed to boost network throughput and system scalability. Moreover, Turbo maintains high system reliability by incorporating an external shuffle service with TCP as a fail-safe option and maintaining metadata management information using the NICs. We have integrated Turbo into Apache Spark and conducted evaluations on both a small-scale test environment and a large-scale cluster with hundreds of server nodes. The findings from the small-scale testbed demonstrate that Turbo enhances network throughput by 15.1% and upholds high system reliability. Additionally, the large-scale production data indicates that Turbo is capable of decreasing job completion times by 23.9% and increasing the job completion rate by $2.03\times $ compared to the current RDMA solutions. In addition, during large-scale tests, we also found that Turbo has improved the computing efficiency of the cluster and saved approximately 24.3% of the CPU utilization.