TrustLift: When Images Make False Claims Look True

Abstract

Multimodal verification systems may assign high confidence to misleading claims when an associated image is visually plausible but does not provide valid evidence. This paper introduces TrustLift, a lightweight framework for detecting such false-reassurance errors and selectively referring risky cases for review. The method compares text–evidence and multimodal predictions, measures visual persuasion lift, and evaluates sensitivity to random and semantically hard image replacements. These signals are combined with uncertainty, similarity, and disagreement features in a harm-aware gate trained under leakage-controlled folds. Experiments on VERITE and AVerImaTeC evaluate counterfactual augmentation, learned gates, non-learned risk baselines, and feature ablations. On VERITE three-way verification, TrustLift increases harmful-error AUROC from 0.628 for confidence-based risk to 0.930 and reduces false reassurance at 90% coverage from 0.181 to 0.134. The results indicate that counterfactual and cross-modal signals improve harmful-error ranking while retaining a CPU-friendly implementation compatible with existing multimodal verifiers.