Multimodal learning combines heterogeneous signals, such as text, audio, and vision, to improve prediction. However, adding modalities does not always help, because modalities may contain useful complementary cues together with noisy, redundant, or contextually unreliable components that interfere with full-modality le...
Ji-Yuan Liu, L. Zheng, W. Zhang et al.· Proceedings of the 35th ACM...· 0 citations
Large vision-language models (LVLMs) routinely use inference-time visual-token compression to prune hundreds of patch tokens down to a compact budget. Existing LVLM backdoors fire whenever a triggered image appears, whereas prior deployment-triggered attacks activate only when model weights are quantized or pruned. How...
Zhi-Fang Zhang, Qi-Rui Huang, Chen-Peng Wang et al.· 0 citations
This survey identifies critical research gaps, including interpretable routing, expert communication, modality integration, modality integration, and lifelong multimodal learning, and positions this survey as a foundation for future research toward interpretable and sustainable multimodal Mixture-of-Experts system.
L. Zheng, Wenhao Liang, W. Zhang et al.· 0 citations
This work audits reading for routing entropy in Attention-Residual variants of Swin-Tiny and DeiT-Small, trained from scratch on CIFAR-10/100 with a soft-binned calibration auxiliary loss, establishing control-dependent gains and incomplete estimator recovery, not the absence of conditional routing information.
Wenhao Liang, Lin Yue, W. Zhang et al.· 0 citations
When a classifier is recalibrated from only a few thousand held-out examples, the capacity of the calibration map becomes a statistical design choice rather than a purely architectural one: a scalar map can underfit structured residual miscalibration, while a highly adaptive map can be hard to estimate reliably from so...
Wenhao Liang, L. Zheng, Lin Yue et al.· arXiv.org· 0 citations
Routing signals of modern vision transformers -- expert gates, attention-residual weights and halting scores -- often improve probes that predict whether the model is correct, and the improvement is commonly read as evidence that routing carries information about errors beyond the model's outputs. We test this inferenc...
Wenhao Liang, Lin Yue, W. Zhang et al.· 0 citations
The proposed Neural Fixed-Parameter Tractable (N-FPT), a neural-model-agnostic framework that uses parameterized algorithms to improve neural solutions without retraining and guide learning, proves this guarantee and global optimality under perfect advice.