A Pyspark-Sparksql Pipeline for Urban Air Quality Monitoring: 4-Year Trends and Who Assessment (2021–2024)

Abstract

Air quality monitoring datasets are inherently temporal, multi-pollutant, and long-horizon in character, making them well suited to scalable distributed computing frameworks that can handle the volume, velocity, and variety of multiyear daily observations while supporting transparent, SQL accessible analytical workflows. This paper presents a reproducible PySpark–SparkSQL analytical pipeline applied to four years (2021–2024) of daily urban air quality monitoring data, addressing five structured research questions about seasonal and temporal variability, pollutant drivers of the Air Quality Index (AQI), human-activity effects on ambient concentrations, and compliance with the revised WHO 2021 Air Quality Guidelines. Our findings document: consistently higher AQI values in winter months, with November peaking at a mean AQI of 342.13 across the four-year period; PM2.5 and PM10 as the dominant AQI drivers with Pearson correlations of 0.87 and 0.91 respectively; marginal but consistent reductions in pollutant levels during holiday and weekend periods, with the strongest effects for trafficrelated NO2; and a chronic, near-daily exceedance of WHO 24-hour mean thresholds for PM2.5 (343–364 exceedance days per year) and PM10 (347–359 days). A previously unreported SO2 exceedance spike in 2024—110 exceedance days compared to fewer than 30 in each prior year—is identified as an anomaly of policy significance indicating an emerging industrial emission source. The pipeline, implemented in Python (PySpark, Pandas, Matplotlib, Seaborn) on Google Colab, provides a reproducible, computationally scalable framework adaptable to other urban air quality monitoring contexts.