Ethical considerations for multimodal artificial intelligence in healthcare

Abstract

Multimodal artificial intelligence (MMAI) is transforming biomedicine by integrating heterogeneous data, e.g., images, speech, behavior, physiological signals, and text, into unified representational spaces. This enables powerful cross-modal inference and data synthesis, with potential gains in diagnostic accuracy, early detection, and patient support. However, these capabilities introduce ethical challenges that exceed existing AI governance frameworks. MMAI can infer sensitive information without patient awareness, and can convert such inferences into new data objects (e.g., images, clinical text) that enter medical records without clear provenance, acquiring the practical status of observed clinical facts. This raises ethical concerns around the infrastructural emedding of inference-based data objects as durable, reusable clinical and research data. The procedures and technical pipelines that govern how such data are classified and integrated into clinical and research infrastructures embed consequential decisions about provenance, attribution, and contestability, often made in advance of adequate governance. In this Perspective, we characterize what distinguishes MMAI-generated data from other forms of algorithmic inference and argue that MMAI is ethically novel in part because it renders cross-modal inferences as recordable data objects, thereby blurring the boundary between observation and generation. We therefore argue for a shift from data-centric protection toward governance of inference and infrastructuring. We propose a four-part agenda: (1) provenance labeling as a prerequisite for accountability; (2) evidence-building to track emergent inference capacities; (3) dynamic consent models responsive to evolving capabilities; and (4) privacy-preserving techniques to limit unjustified or unconsented inferences. These steps aim to support innovation while safeguarding individual rights and expectations.