Figure 2 from Counterfactual Diffusion Models Provide Interpretable Explanations of Artificial Intelligence Models in Pathology

Abstract

Results for counterfactual image generation for the CRC-VAL-HE-7K dataset. A, Tissue-type classifier-guided counterfactuals of a dysplastic tile, generated by shifting its feature vector toward the predicted healthy colon mucosa class. Changes reflect what the model requires to flip the prediction, with increasing manipulation amplitude shown above each image. B, The t-SNE projection of the feature extractor’s latent space (training set), colored by class: adipose (ADI), background (BACK), debris (DEB), lymphocytes (LYM), mucus (MUC), smooth muscle (MUS), healthy colon mucosa (NORM), cancer-associated stroma (STR), colorectal adenocarcinoma epithelium/dysplastic tissue (TUM). The features of dysplastic tile in A, initially near the boundary between TUM (aqua) and NORM (gray), shift toward the healthy cluster (pink points in the zoomed-in plot; gray area—arbitrary healthy mucosa zone) as manipulation amplitude increases. C, Morphologic feature prevalence in original and counterfactual image pairs. Horizontal bars show the percentage of image pairs in which at least one of four raters (three board-certified pathologists and one pathology trainee) identified each morphologic feature as present in the original (dark gray) or counterfactual (light gray) image. Right, Mean pairwise inter-rater agreement (Cohen κ) per feature, computed across all pairs and both directions combined. D, Counterfactuals of a healthy mucosa tile shifted toward dysplastic epithelium. Below, Pixel-level difference maps (darker regions indicate greater changes), showing the most change in gland regions, and SSIM values, which quantify the similarity between the original and manipulated tiles. Bottom, Segmented and classified nuclei: pink (epithelial), orange (connective tissue), blue (plasma), deep purple (lymphocytes), aqua (neutrophils), and green (eosinophils). E, Tile-level differences in cell type fractions between (left) real NORM tiles (n = 741) and synthetic (counterfactual images generated from TUM tiles, n = 1,233; full statistical details are provided in Supplementary Table S4). Right, Tile-level differences in cell-type fractions between real TUM tiles (n = 1,233) and synthetic (counterfactual images generated from NORM tiles, n = 741).