A scalable hybrid recommendation framework for e-commerce using distributed big data processing

Abstract

The online retail platforms of the last decade have gone through an exponential expansion, with Amazon being a prime example. This expansion has resulted in the generation of a vast amount of user–product interaction data, which traditional single-node systems struggle to process effectively. This research presents a scalable, deployment-oriented e-commerce recommendation system that uses Apache Spark, Hadoop, Pig, Hive, and Mahout within a Dockerized big data ecosystem. The pipeline integrates collaborative filtering using Alternating Least Squares (ALS) and content-based filtering using TF–IDF into a hybrid recommendation engine, and is automated end-to-end for ingestion, preprocessing, modeling, and evaluation, with a Flask web interface for monitoring and configuration. The system is evaluated on Amazon review datasets ranging from a 4.6-million-rating modeling subset to a full ingestion corpus exceeding 40 million records. On a real three-node Apache Spark/Hadoop cluster (HDFS replication factor 3), we empirically demonstrate sub-linear growth of training time with data volume, a measurable multi-node speedup, and fault tolerance under a deliberately induced node failure. The hybrid model improves recommendation quality over the standalone models, attaining an RMSE as low as \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx 0.94$$\end{document} and Precision@10 up to \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\approx0.74$$\end{document} across the evaluation datasets. This research provides a comprehensive account of the design, implementation, evaluation, and future directions for scalable big data analytics in recommendation systems.