Potential risks of generative artificial intelligence in kidney care

Abstract

Generative artificial intelligence (GenAI), particularly large language models, is rapidly emerging as a transformative tool in medicine, including nephrology, with applications spanning clinical decision support, documentation, patient education, and research. Unlike traditional machine learning systems, GenAI produces probabilistic, context-dependent outputs, introducing novel opportunities but also distinct risks. This review examines key risk domains associated with the integration of GenAI into kidney care. These include data-related challenges such as bias, limited representativeness, and data quality issues; technical limitations such as hallucinations and prompt-dependent variability; and cognitive risks, including automation bias and overreliance. The differences between human clinical reasoning and AI-generated outputs further complicate safe implementation, particularly in high-stakes decision-making contexts. In addition, broader systemic concerns are addressed, including inequities in access, linguistic and infrastructural disparities, and the environmental footprint associated with large-scale AI systems. We highlight that these risks are not isolated but interconnected, with potential implications for patient safety, clinical decision-making, and health system equity. While GenAI may enhance efficiency and support clinical workflows, its outputs require critical appraisal and validation within the clinical context. Safe and responsible implementation will depend on a combination of technical safeguards, structured prompting, rigorous validation, and human-in-the-loop oversight. Regulatory frameworks provide an essential foundation, but clinical accountability remains central. Ultimately, the integration of GenAI in nephrology should prioritize safety, equity, and sustainability, ensuring that technological innovation translates into meaningful and responsible improvements in patient care.