When Prompts Ignore Structure: Graph-Based Attribute Reasoning for Calibrated VLMs

Tanay Sodha Aditya Sharma Ramya Hebbalaguppe Vinti Agarwal Pranav Murthy Yeluripaty

Jul 2026 · arXiv.org · Vol abs/2607.07395 · 0 citations · 26 references

Computer Science

TL;DR

This work proposes ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies.

Abstract

Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.

DOI arXiv Semantic Scholar

View source

Similar papers

#artificial intelligence Preprint Sep 2026

DocHop: Benchmarking Out-of-domain Multi-hop Reasoning in Information-Dense Documents

This work introduces DocHop, a benchmark for integrated chart--context reasoning in document-style images and constructs DocHop via a stochastic logic-first generation pipeline with controllable reasoning depth and visual density, to enable systematic evaluation.

Zhuoran Yu, Le Thien Phuc Nguyen, Jaden Park et al. · 1 citation

Jul 2026

Similarity Is Not Logic: Factored Inference for Dual-Encoder Vision-Language Models

Despite-encoder vision-language models expose a similarity interface that enables zero-shot retrieval but fails compositional constraints, this work proposes factored inference, which separates evidence extraction from constraint execution, and introduces LCSE (Logic-Constrained Score Editing), a training-free method t...

S. Alshehri, Zhan-Tao Yang, Han Zhang et al. · arXiv.org · 0 citations

#machine learning Preprint Sep 2026

ABSOL: Aggregated Bayesian Subsampling Orchestrated with LLMs

A hybrid LLM-guided Bayesian network structure-learning framework that uses LLMs as bounded semantic guides, and shows that language-derived semantic knowledge can substantially improve scalable probabilistic structure learning when used as bounded guidance within a statistically grounded reasoning pipeline.

Jackson Hassell, Chen Shen, Estevam Hruschka · 0 citations

#machine learning Preprint Aug 2026

HalluPrism: When Multimodal Uncertainty Should Diagnose, Not Decide

HalluPrism, a behavioral diagnostic that re-runs an answer after visual degradation, blank-image replacement, and grounding or relation checks is proposed, a behavioral diagnostic that separates failure diagnosis from abstention scoring.

Aman Prakash, Sourish Dasgupta, Tanmoy Chakraborty · 0 citations

Preprint Aug 2026

EdgeLM: Edge Demonstrations for Language Models'Table Understanding

Across five data wrangling tasks, fifteen datasets, and five open-weight and proprietary LLMs, EdgeLM consistently achieves the best or near-best performance in every setting, while ablations show that the two forms of edge evidence provide complementary benefits.

Soroush Omidvartehrani, Mohammadamin Habibollah, Mohammadreza Daviran et al. · 0 citations

#machine learning Preprint Aug 2026

Who Should Teach? Confidence-Aware Dual-Teacher Learning for Few-Shot Node Classification on Text-Attributed Graphs

This work proposes CoTeach, a Confidence-aware dual-teacher learning framework that dynamically selects the more reliable teacher for each node, and demonstrates that CoTeach consistently improves few-shot node classification performance while reducing unnecessary LLM utilization and associated monetary costs.

Hojin Kim, Sujin Yoon, Sung-Su Lim et al. · 0 citations