Non-invasive fairness drift monitor for machine learning models using conformance constraints

Abstract

Machine learning models deployed in consequential domains can become unfair toward protected subgroups as the data they receive drifts over time, yet the protected attributes needed to measure fairness directly are often unavailable at runtime due to privacy regulation and operational constraints. This creates a gap: existing fairness toolkits require protected labels and perform one-time audits, while generic drift detectors monitor continuously but cannot localize which subgroup a shift harms. This thesis develops a non-invasive fairness drift monitor that addresses this gap by repurposing Conformance Constraints, a data-profiling primitive, as a temporal fairness signal. The monitor learns per-subgroup distributional profiles at baseline, using protected attributes only once, and thereafter tracks violation of those profiles over incoming data batches without any runtime access to protected attributes. Across three fairness benchmarks, two classifiers, and nineteen controlled drift scenarios, a minority-subgroup conformance-constraint signal tracks fairness degradation more closely than KS and KL detectors under global drift on the two datasets where it is defined, while the conformance constraint signals defined on all three datasets remain competitive with those baselines under both regimes. Subgroup localization provides its clearest advantage under global drift, where a minority-subgroup signal substantially outperforms both aggregate signals and the baselines. The correlations are modest in absolute terms, indicating that the monitor functions as a screening instrument that flags fairness degradation for closer investigation rather than as a precise estimator. The approach offers privacy-preserving, subgroup-aware fairness monitoring suited to regulated deployment settings.