A Metadata-Driven Framework for Automated Bronze-to-Silver Data Ingestion, Validation and Loading on Databricks

Abstract

Bronze-to-Silver transformation is a critical and repetitive stage of the Databricks medallion architecture, where raw, ungoverned data must be cleaned, type-cast, validated, and merged into curated Delta tables. In most production environments this stage is implemented through per-table notebooks or jobs, which leads to duplicated logic, inconsistent quality enforcement, and a slow, error-prone onboarding process for new datasets. This paper presents a metadata-driven framework that automates the Bronze-to-Silver pipeline on Databricks. A central metadata repository defines source-to-target column mappings, target data types, default and transformation rules, incremental (delta, i.e., only the records that are new or changed since the last run) watermark columns, load strategy (append or upsert), and a set of declarative data-quality (DQX-style) rules such as null checks, date-format checks, and alphanumeric checks. At runtime, the framework reads the active metadata, fetches full or incremental data from Bronze, evaluates the configured data-quality rules against every column, splits records into valid and quarantined data frames, performs type conversion and column projection, and writes the resulting valid records into Silver Delta tables using append or merge (upsert) semantics. We describe the system architecture, the metadata model, and the three core process flows—data fetch, rule-based validation, and data conversion/load—and outline an evaluation plan covering engineering effort, data-quality outcomes, and performance. The approach removes hand-written, table-specific ETL code from the Bronze-to-Silver stage and replaces it with a governed, auditable, and reusable configuration layer.