Speculative decoding accelerates autoregressive generation by allowing a lightweight drafter to propose future tokens while a target model verifies them in parallel. Its lossless guarantee has motivated a line of work that pushes the drafter itself toward parallel generation. The most recent paradigm is block-parallel generative drafting, including diffusion-based methods such as DFlash and DSpark, achieving up to 3.6x speedup on common daily chatting tasks. While this transition is well studied in text-only LLMs, its applicability to multimodal models remains an open question. Existing multimodal speculative decoding efforts focus on input compression, adapter alignment, candidate coverage, or modality-specific verification; however, block-parallel generative drafting remains largely unexplored. To bridge this gap, this paper combines a modality-centered survey with a cross-architecture empirical study to ask: Is multimodal speculative decoding ready for diffusion-based parallel drafting? In this survey, we systematically analyze a wide spectrum of multimodal models, spanning Vision-Language, Video-Language, Audio, and Vision-Language-Action (VLA) architectures, from the dual perspectives of drafting parallelism and cross-modal information interaction. We introduce a unified taxonomy that isolates drafter-side parallelism from orthogonal design choices such as tree construction and verification strategies. Furthermore, we provide a comprehensive empirical comparison of existing methods under varying degrees of parallelism across standardized multimodal benchmarks, including OCR, VQA, visual reasoning, and image captioning. Finally, we summarize the limitations of current approaches, discuss open challenges, and outline promising future directions for this rapidly evolving field.
Yantao Li, Huanlin Gao, Fang Zhao et al.· 0 citations
Existing Multimodal Large Language Models (MLLMs) predominantly rely on image-text pairs for modality alignment pretraining, mapping global image representations to long textual descriptions. However, this image-level alignment suffers from referential ambiguity: models struggle to infer the correspondences between multiple visual objects and textual entities from the global representation, leading to data inefficiency and suboptimal semantic grounding. To address this, we propose MultiModal Code-Switching (MMCS), a novel pretraining paradigm that provides explicit object-level supervision. Inspired by the linguistic phenomenon of code-switching, MMCS interleaves vision and language by replacing textual entities with their corresponding visual objects, enforcing local vision-language grounding. We further develop a scalable data synthesis pipeline to generate a pretraining dataset of 773K samples with accurate object-entity correspondences. Experiments show that MMCS is highly data-efficient: with only 50K samples, it matches or surpasses models trained on 600K image-text pairs. Furthermore, MMCS consistently improves visual grounding and perception capabilities across varying model scales.
Changhao Xiang, Shangyu Xing, Zhen Wu et al.· 0 citations
Antigen-specific antibody retrieval aims to rank candidate antibodies for a target antigen, providing an early virtual-screening step before structural modeling or experimental validation. Existing sequence-based antibody-antigen interaction studies often formulate the problem as pairwise binding prediction, and random or non-clustered evaluations can over-estimate generalization when related antigens appear across training and test data. We study a strict antigen-cluster out-of-distribution (OOD) retrieval setting in which test antigens come from sequence clusters unseen during training. This setting is difficult because binding is driven by local epitope-CDR complementarity, while available databases mainly contain observed positive complexes and lack reliable negative labels for unlabeled candidates. We propose Ab-CASLR, an antibody CDR-aware slot late-interaction retriever that encodes antigens with ESM-2, encodes antibodies with IgBert, constrains antibody-side latent slots to complementarity-determining regions (CDRs), and scores local slot compatibility instead of single-vector global similarity. On a strict OOD benchmark with 849 antigen queries and 869 candidate antibodies, the model achieves 7.42% Hits@10, outperforming k-mer homology transfer at 5.53% Hits@10 and yielding 6.28-fold enrichment over exact random screening at K = 10. Ablations and diagnostics show that CDR-constrained antibody slots remain diverse, whereas antigen-side latent slots collapse into similar summaries. These results support CDR-aware local antibody representation as a useful inductive bias for early binder recovery under strict OOD evaluation, while antigen-side epitope grounding remains unresolved.
Peishuo Liu, Mianzhi Pan, Chenyang Yan et al.· bioRxiv· 0 citations