Comparative analysis of big data processing tools on the Hadoop platform

Abstract

Objective. This article provides a comprehensive comparative review of the leading tools used for processing big data in the Apache Hadoop ecosystem, with the aim of providing organizations with criteria for selecting the optimal solution for different types of workloads. The relevance of this research is driven by the fact that modern businesses require the analysis of data of all shapes and sizes, including structured and unstructured data, in order to gain a better understanding of their products, customers, and markets. The increasing volume, velocity, and variety of data (Big Data) has led to the need for specialized platforms to handle it. The Apache Hive instability issue is a key architectural challenge, as Hive is the most resource-intensive when performing identical tasks with Spark, as it starts processing input data without pre-analyzing it. Method . A review of sources covering a wide range of tools and architectural solutions for processing big data in the Apache Hadoop ecosystem and beyond. The review included specialized scientific articles, technical guides, experimental research results, and industry analysis reports. The proposed methodology combines a comparative analysis of existing approaches and a comprehensive methodology for choosing optimal tools and architectural solutions for processing big data. Result . The analyzed characteristics of Cloudera Impala and Apache Hive allow us to conclude that these SQL tools for analytical data processing, in the Hadoop ecosystem, complement each other. Conclusion . It is premature to completely replace Apache Spark with Flink due to the potential need for additional infrastructure, but Flink is a viable solution for tasks that require minimal response time.