Word Count Frequency Statistics Application of MapReduce Based on Alibaba Cloud FC

Abstract

. The core part of distributed text processing is the MapReduce framework. Alibaba Cloud Function Compute and Object Storage Service can realize its serverless deployment, but the parameter matching rules for data sharding and memory configuration still stay unclear. This study builds a MapReduce distributed architecture based on Alibaba Cloud FC+OSS, and uses English text as test data. Therefore, three groups of controlled variable experiments are designed for exploring the influence of shard size and memory configuration on word frequency counting efficiency. Thus, experiments prove the architecture’s feasibility with 100 percent accuracy, and find out the optimal parameter combination is 50MB shards + 1GB memory + 4 concurrency levels—this combination can get 20 percent efficiency improvement without extra costs. Hence, this study makes clear the shared-memory matching relationship, and provides experimental references for parameter configuration in the distributed data processing of Alibaba Cloud FC.