Competence Is Not Protection: Prompt-Channel Demographic Bias in Pathology Vision-Language Models, and the Geometry That Predicts It

Abstract

Part IV of a series on equity and reliability in medical imaging AI. Part III of this series showed that naming a demographic group in the text prompt of a medical vision–language model changes its diagnosis on a fixed chest radiograph, that the perturbation is exactly rank one in the standard positive/negative pair readout, and that group-specific thresholds therefore provably cannot remove it. Two questions were left open: whether any of this survives a change of imaging domain, and whether it matters for a model that is actually good at its task — Part III's strongest encoder reached a mean AUC of 0.687. This paper audits four pathology vision–language models (PLIP, QuiltNet-B-32, CONCH, and the slide-level TITAN) under the descriptor set from Part III, unchanged, and combines them with Part III's three radiology encoders for a seven-model comparison spanning two imaging domains and both patch and slide granularity. On TITAN the audit runs at full strength: 8,010 whole-slide images from 6,582 patients across 25 TCGA cancer types, every interval clustered on patient, mean neutral-prompt AUC 0.982 with all 25 projects clearing the utility floor. Four results. The rank-one identity transfers exactly — verified to 5.2×10−8 on real slide embeddings, two orders of magnitude tighter than Part III because TITAN has no learned logit scale to amplify float error. Part III's threshold-irremovability corollary therefore holds for whole-slide pathology. Diagnostic competence does not protect against prompt-channel bias. Across seven encoders, neutral-prompt AUC does not predict the effect (r = −0.443, p = 0.32): CONCH at AUC 0.849 shows 21.1 points of mean absolute excess false-negative rate, indistinguishable from OpenAI CLIP at AUC 0.518. What predicts it is a purely geometric quantity, the descriptor perturbation measured against the diagnostic contrast (r = +0.780, p = 0.039), reported as a hypothesis with n = 7 rather than a law. The model answers the token and not its truth, including for race. Part III could test congruence only for sex and age, because NIH ChestX-ray14 records no race; TCGA does, and the difference-in-differences estimator gives a median 0.0045 logits for race and 0.0010 for sex, against a specimen-site positive control 164–730× larger in 4 of 4 pairs at tighter precision. Anatomically impossible descriptors — "a male patient showing endometrial carcinoma" — move the score as much as coherent ones. Prompt symmetrisation is a domain-dependent intervention rather than a general one: it removes 59–63% of the effect on radiology encoders, 26–34% on pathology patch encoders, and amplifies the effect by 94% on TITAN. The text-side symmetry residual separates the two domains completely (Mann–Whitney p = 0.029). The practical consequence is a correction to how this series has framed mitigation. A deployer cannot infer prompt-channel robustness from a model's accuracy, and cannot assume a mitigation validated in one imaging domain transfers to another — including on the strength of Part III. Honest scope. Unlike Parts II, IIb, and III, this part is not preregistered; PROTOCOL_PART4.md is retrospective and says so in its first line, recording which hypotheses preceded their endpoints and which did not. Only TITAN carries interval estimates: the public CRC-VAL-HE-7K mirror records no patient identifiers, so the three patch-level encoders are reported as point estimates only. The anatomically-impossible result rests on five cells and is a demonstration of mechanism, not an estimate of magnitude. This paper also supplies the out-of-domain evidence behind amendment 8 to Part III's protocol, which withdrew Part III's proposed text-only predictor of symmetrisation failure. Files. The manuscript PDF (21 pp, 4 figures); the retrospective protocol; and an artifact archive containing all 13 analysis scripts and every result table as CSV/JSON. Model weights are not redistributed — TITAN, CONCH, and UNI are CC-BY-NC-ND-4.0 and were obtained through the gated Hugging Face process. Ethics. No patient data was collected. TCGA is a public consortium dataset; CRC-VAL-HE-7K is public and de-identified. This is an audit of a model property, not a clinical study, and makes no claim about patient outcomes.