Training Dataset Exposure Through Adversarial Model Query Analysis: A Black-Box Privacy Breach Framework

Abstract

Machine-learning-as-a-service (MLaaS) platforms let organizations deploy predictive models without revealing their internal architecture, but this opacity does not guarantee the privacy of the data used to train them. This paper presents a systematic framework for membership inference attacks, in which an adversary with only black-box query access to a deployed classifier determines whether a specific individual's record was part of its training set. We introduce a shadow-model methodology that trains auxiliary classifiers on synthetically generated data produced via model-based synthesis, known population statistics, or noisy real data to learn the behavioral differences a target model exhibits on seen versus unseen inputs. Evaluated against commercial platforms including Google Prediction API and Amazon ML across seven datasets spanning healthcare, retail, location, and image domains, our attacks achieve median precision of 0.657 and 0.678 on the two platforms, respectively, with healthcare records showing vulnerability of approximately 65.7% precision. We identify overfitting, output granularity, and class imbalance as primary drivers of leakage, and evaluate mitigations such as output truncation, temperature scaling, and regularization, finding that each offers only partial protection at the cost of utility. These findings expose significant gaps in current regulatory frameworks such as GDPR and HIPAA regarding inference-based privacy violations.