NoEsis: A Modular LLM with Differential Privacy and Knowledge Transfer
TL;DR
NoEsis integrates differential privacy with a hybrid architecture that combines domain-specific adapters, acting as experts, and common prompt tokens, acting as a knowledge-sharing backbone, and empirically show protection against Membership Inference Attacks.
Abstract
Large Language Models (LLMs) are typically trained on vast amounts of data, springing from various sources. Even when designed modularly, such as Mixture-of-Experts models, LLMs can leak private information on their source training data. Conversely, training such large models in isolation hinders generalization and does not allow for the sharing of knowledge. Therefore, we propose a framework, NoEsis, which builds upon the desired properties of modularity, privacy, and knowledge transfer. NoEsis integrates differential privacy with a hybrid architecture that combines domain-specific adapters, acting as experts, and common prompt tokens, acting as a knowledge-sharing backbone. The results from our evaluation on CodeXGLUE show that NoEsis can achieve provable privacy guarantees with knowledge transfer across domains, and empirically show protection against Membership Inference Attacks. On code completion tasks, NoEsis bridges 84% of the accuracy gap between a non-shared and a non-private baseline. This work addresses the growing need for privacy-preserving AI systems that can operate across institutional boundaries while respecting data sovereignty and preventing unauthorized knowledge extraction.