Part IV of a series on equity and reliability in medical imaging AI. Part III of this series showed that naming a demographic group in the text prompt of a medical vision–language model changes its diagnosis on a fixed chest radiograph, that the perturbation is exactly rank one in the standard positive/negative pair readout, and that group-specific thresholds therefore provably cannot remove it. Two questions were left open: whether any of this survives a change of imaging domain, and whether it matters for a model that is actually good at its task — Part III's strongest encoder reached a mean AUC of 0.687. This paper audits four pathology vision–language models (PLIP, QuiltNet-B-32, CONCH, and the slide-level TITAN) under the descriptor set from Part III, unchanged, and combines them with Part III's three radiology encoders for a seven-model comparison spanning two imaging domains and both patch and slide granularity. On TITAN the audit runs at full strength: 8,010 whole-slide images from 6,582 patients across 25 TCGA cancer types, every interval clustered on patient, mean neutral-prompt AUC 0.982 with all 25 projects clearing the utility floor. Four results. The rank-one identity transfers exactly — verified to 5.2×10−8 on real slide embeddings, two orders of magnitude tighter than Part III because TITAN has no learned logit scale to amplify float error. Part III's threshold-irremovability corollary therefore holds for whole-slide pathology. Diagnostic competence does not protect against prompt-channel bias. Across seven encoders, neutral-prompt AUC does not predict the effect (r = −0.443, p = 0.32): CONCH at AUC 0.849 shows 21.1 points of mean absolute excess false-negative rate, indistinguishable from OpenAI CLIP at AUC 0.518. What predicts it is a purely geometric quantity, the descriptor perturbation measured against the diagnostic contrast (r = +0.780, p = 0.039), reported as a hypothesis with n = 7 rather than a law. The model answers the token and not its truth, including for race. Part III could test congruence only for sex and age, because NIH ChestX-ray14 records no race; TCGA does, and the difference-in-differences estimator gives a median 0.0045 logits for race and 0.0010 for sex, against a specimen-site positive control 164–730× larger in 4 of 4 pairs at tighter precision. Anatomically impossible descriptors — "a male patient showing endometrial carcinoma" — move the score as much as coherent ones. Prompt symmetrisation is a domain-dependent intervention rather than a general one: it removes 59–63% of the effect on radiology encoders, 26–34% on pathology patch encoders, and amplifies the effect by 94% on TITAN. The text-side symmetry residual separates the two domains completely (Mann–Whitney p = 0.029). The practical consequence is a correction to how this series has framed mitigation. A deployer cannot infer prompt-channel robustness from a model's accuracy, and cannot assume a mitigation validated in one imaging domain transfers to another — including on the strength of Part III. Honest scope. Unlike Parts II, IIb, and III, this part is not preregistered; PROTOCOL_PART4.md is retrospective and says so in its first line, recording which hypotheses preceded their endpoints and which did not. Only TITAN carries interval estimates: the public CRC-VAL-HE-7K mirror records no patient identifiers, so the three patch-level encoders are reported as point estimates only. The anatomically-impossible result rests on five cells and is a demonstration of mechanism, not an estimate of magnitude. This paper also supplies the out-of-domain evidence behind amendment 8 to Part III's protocol, which withdrew Part III's proposed text-only predictor of symmetrisation failure. Files. The manuscript PDF (21 pp, 4 figures); the retrospective protocol; and an artifact archive containing all 13 analysis scripts and every result table as CSV/JSON. Model weights are not redistributed — TITAN, CONCH, and UNI are CC-BY-NC-ND-4.0 and were obtained through the gated Hugging Face process. Ethics. No patient data was collected. TCGA is a public consortium dataset; CRC-VAL-HE-7K is public and de-identified. This is an audit of a model property, not a clinical study, and makes no claim about patient outcomes.
Omar Mohammed· Zenodo (CERN European Organi...· 0 citations
Parts I and II of this series audited and attempted to remediate subgroup disparities in chest radiograph(CXR) classifi ers, and Part II declared an explicit limitation: multimodal systems combining imageswith clinical text were out of scope, because “text-derived features may carry demographic informationmore directly than images.” This paper closes that gap. We audit BiomedCLIP, an open-weightbiomedical vision-language model, on 25,596 radiographs from 2,797 patients in the offi cial NIHChestX-ray14 test split, and ask a question that observational subgroup audits structurally cannot: whathappens to the diagnosis when the patient’s demographics are stated in thepromptwhile the image, themodel, the label, and the decision threshold are all held fi xed?The answer is that the prompt is a diagnostic input. Naming a demographic group in the text changesthe false-negative rate by 15.5 percentage points on average and by up to 73.8 points in the worst cell,fl ips the binary call on a median 14.1% of positive cases, and degrades AUC by up to 0.137 — the last ofwhich matters because a threshold cannot change AUC, so that component is not an operating-pointartifact and no post-hoc correction can absorb it. Every one of these fi gures is reported as excess over abank of content-free qualifi ers (“a hospital patient”, “a patient referred for imaging”), which weintroduce as a necessary control: the format eff ect alone produces an apparent gap of 8.9 points,comparable to race’s 12.0, so an audit lacking this control would attribute most of a grammatical artifactto demography.We replicate on two further models spanning a domain-specifi city axis — PubMedCLIP (radiologycaptions) and OpenAI CLIP (general web) — and the replication both strengthens and corrects theaccount. The eff ect appears in all three, in 365 of 375 cells atq< 0.05, and it islargest in OpenAI CLIP,which detects no fi nding above chance: 28.9 points of excess FNR and a 30.3% fl ip rate from a modelwith no radiographic competence. Prompt-channel bias is therefore not a model applying clinicaldemographic priors; it is a property of contrastive image–text pretraining with a pair readout. Againstthat, the ordering across descriptor families doesnotgeneralise — socioeconomic descriptors areBiomedCLIP’s second-largest family and PubMedCLIP’s smallest — so we report that as BiomedCLIP-specifi c rather than as a property of medical VLMs. Age descriptors dominate in all three.Three results explain and constrain the eff ect. First, for the standard positive/negative prompt-pairreadout the perturbation isexactly rank one— verifi ed to 4.6 × 10⁻⁶ across 420 (fi nding × descriptor)cells in every one of the three models — so it is a single fi xed direction independent of the image andtherefore not indexed by the patient’s true group. This places prompt-channel biasupstreamof everydecision-rule remedy in Part II’s stage taxonomy: group-specifi c thresholds provably cannot remove it.Second, the eff ect decomposes into an image-independent component that behaves like an uncontrolledthreshold off set and an image-specifi c component that re-ranks patients; the latter is 26–43% of themean-square shift and is irreducible. Third, because the design is paired at the image level, itsminimum detectable eff ect is 3.4 points against 9.4 for an equivalent observational audit — anobservational study would need roughly 5.9× more positive cases — which dissolves, for this class ofbias, the audit-power obstacle Part II quantifi ed.A positive control validates the congruence null. Section 7 fi nds that the model responds to a stated sexbut essentially not to whether it is true, which is only meaningful if the estimator can detect evidenceuse at all. Substituting view position — recorded in the metadata and plainly visible in the radiograph— yields a diff erence-in-diff erences 16.9× larger, signifi cant in 11 of 11 fi ndings against 1 of 11 for sex, atgreater precision. The sex null is substantive, not a power failure. For mitigation we compare prompt symmetrisation, which inserts the descriptor into both prompts ofthe pair, against the orthogonal and calibrated text-side projections of Chuang et al. On BiomedCLIPsymmetrisation reduces mean absolute excess FNR from 15.5 to 5.2 points at no utility cost, while bothprojections reach only 8.4–9.5 points and cost 6–7 AUC points. We are explicit that symmetrisation isthe zero-cost degenerate limit of Chuang et al.’s calibration objective rather than a new idea. It also doesnot always work: it reduces the eff ect by about 60% on BiomedCLIP and OpenAI CLIP but is inert onPubMedCLIP. We proposed, and Section 10.4 withdraws, a text-only statistic intended to predict thatfailure in advance: it is contradicted by OpenAI CLIP within these same results, and by twohistopathology encoders out of domain. Whether symmetrisation will work must therefore bemeasured on the model in question, which is cheap but not free. We conclude that any deploymenttemplating patient metadata into a promptable diagnostic model has introduced a bias channel that itsimage-side audit cannot see and its threshold policy cannot fi x.
Omar Mohammed· Zenodo (CERN European Organi...· 0 citations