A review of bias detection and fairness auditing techniques in LLMs

Abstract

The rise of large language models (LLMs) has sparked worries about inherent social biases and issues related to fairness. Earlier studies have investigated bias identification in word embeddings, interventions aimed at fairness in algorithms, and frameworks for auditing at the system level. Nonetheless, these methods remain disorganized, with variations in datasets, evaluation methods, and implementation processes. In this paper, we provide a thorough literature review to encapsulate prior research on bias identification and fairness auditing, categorizing the findings according to various stages of study. Additionally, we analyze the limitations in coverage and consistency of widely used benchmark datasets. To tackle these issues, we propose a unified pipeline for dataset integration and a modular framework for bias auditing. Recognized significant research gaps include the absence of intersectional bias modeling, a shortage of standardized evaluation metrics, and challenges in scalability for real-time auditing systems.