While Vision-Language Models (VLMs) have demonstrated remarkable performance in processing and understanding both text and images, their large parameter sizes lead to significant computational overhead, limiting their deployment on resource-constrained devices. While pruning has been effective for compressing Large Language Models (LLMs), directly applying it to VLMs leads to significant performance drops, largely due to redundant visual tokens interfering with importance estimation. To this end, we propose SlimVLM, a structured pruning framework designed to compress VLMs while preserving their task performance. We introduce an adaptive visual token selection strategy for VLMs that leverages average text-to-visual attention scores to assess the importance of visual tokens, removing redundant ones during pruning based on a set threshold, thereby optimizing the importance calculation. Recognizing the varying tolerance to sparsity across different modules, we also propose a Sensitivity-aware dynamic pruning mechanism that determines the appropriate pruning ratio for each module by calculating the linear reconstruction error between the outputs of the pruned and unpruned modules, ensuring overall performance stability. Experimental results show that SlimVLM outperforms existing methods across multiple multimodal benchmarks, achieving state-of-the-art performance.
Yaozhi Wen, Jialong Guo, Zhenliang Ni et al.· 0 citations
Semantic segmentation is critical for intelligent robotics to understand complex environments. While CNN-based models on RGB images achieve high performance, their accuracy drops in fast-motion or low-light scenes. Fortunately, event cameras, with high temporal resolution and low latency, offer robust perception in such challenging conditions. Many event-image fusion methods attempt to combine the complementary strengths of both modalities, but most adopt simple fusion strategies without considering intermodal correlations or designing computationally expensive architectures, resulting in degraded accuracy and high energy costs. To overcome these limitations, we propose a lightweight spiking neural network (SNN)-based event-image fusion network (Spike-EIFNet) that leverages the complementary strengths of multimodal fusion and energy-efficient spike-driven computation. In particular, to reduce computation cost for lightweight, Spike-EIFNet adopts a dual-branch SNN encoder to process events and images in parallel. Then, to improve the segmentation accuracy with enhanced feature interaction, we introduce a spike-driven cross-modal fusion (SCMF) module, consisting of a modality-aware fine-grained extraction (MFE) stage to capture dynamic cues from events and spatial details from images, followed by a cross-modal interaction and fusion (CIF) stage for effective feature alignment. Finally, a lightweight feature enhancement (LFE) module is proposed to further refine feature representations and facilitate deep-shallow feature fusion. Extensive experiments demonstrate that Spike-EIFNet achieves 67.34% and 58.09% mean intersection over union (mIoU) on the DDD17 and DSEC-Semantic datasets while consuming $72.83\times $ and $100.26\times $ less energy, respectively. Compared with ANN-based methods, Spike-EIFNet significantly reduces energy consumption; among SNN-based methods, it achieves the highest segmentation accuracy with a favorable accuracy-efficiency tradeoff. Code is available at: https://github.com/Chensyfighting/Spike-EIFNet.
Siyu Chen, Qie Liu, Xianlei Long et al.· IEEE Transactions on Neural...· 0 citations