ReFyn: An Automated ETL Pipeline for Heterogeneous Data Processing and Analytics

Abstract

Modern day software products based on data usually rely on spreadsheet datasets which might be poorly structured and dirty. The current paper outlines the creation of a pipeline automation system called ReFyn, which is aimed at the effective transformation of diverse data formats such as CSV files and Excel sheets. Data cleaning, standardization, and feature engineering have been enabled through a modular design based on Apache Airflow, Flask, and PostgreSQL. Automation mapping will help ReFyn tackle issues like varying naming convention, absent columns, differences in encoding, and discrepancies in categories. This process facilitates easy data processing because everything goes according to plan. From the experiments that have been performed, it is clear that the framework offers a way to achieve easy data processing.