SRFE: Measuring Chain-of-Thought Faithfulness with Independent Stepwise Visual Probes

Abstract

Using stepwise probe as an external reference or Chain-of-Thought (CoT) to evaluate visual-language model (VLM) analysis ability is common, but the faithfulness of PASS/FAIL verdicts in VLM produced by explicit CoT and independent SRFE probe needs further work to figure out. Prior works on CoT faithfulness are mainly about large language model (LLM) and internal interpretability signals, with limited VLM, Vision Prompt Distillation (VPD), structured verdict settings and behavioral probes. This paper focuses on evaluating Stepwise Reasoning Faithfulness Evaluation (SRFE), a five-step protocol (object->color->size->relation->composition) with prompt-defined order, and CoT faithfulness after distillation, trying to align them using probe Supervised Fine-Tuning (SFT) to reduce two different methods to a single faithfulness evaluating method. To get features of SRFE, the paper compares the independent and conditional probe version and find weak chain effect in model then order ablation experiment find SRFE is order invariance. After VPD distillation on Qwen2-VL, teacher and student have a high agreement (97.6% stepwise, 88% exact bits), supporting SRFE probe is procedure faithfulness but not internal reasoning faithfulness. To find Chain-of-thought procedure faithfulness, the paper uses VLM to generate explicit CoT then separate the answer and compare each one under SRFE to get the similarity, finds that baseline CoT-SRFE probe partial misalignment. To improve alignment, SRFE probe supervised SFT on CoT is implemented, but finds a trade-off between independent probe agreement, chain consistency, and say-do &do-say.