A Lightweight API Gateway for Interactive Open Data Portal Analysis

Abstract

Researchers and developers face significant barriers when conducting systematic analyses across multiple open data portals or building tools that interact with diverse portal infrastructures. While production-grade systems like Piveau and ckanext-harvest serve portal operators for large-scale metadata harvesting, they require substantial setup and are unsuitable for ad-hoc research exploration. This paper presents a lightweight API gateway specifically designed for interactive, research-oriented portal analysis. The system auto-detects portal software (CKAN, DKAN, Piveau, Socrata, OpenDataSoft), exposes a unified REST API grounded in DCAT-AP, and provides live metadata quality assessment. Built on FastAPI for async-first performance, the gateway serves as the Data Integration Layer in our previously published AI-enhanced open data framework, enabling applications such as automated accessible dataset descriptions. The architecture employs an adapter pattern with pluggable connectors, implements link-following for availability verification in federated networks, and supports multi-portal comparative analysis. We position this work as complementary to existing production infrastructure, targeting lightweight workflows for systematic portal studies and rapid prototyping of data-driven applications.