CIS-Fuse is proposed, a spiking network that performs cross-modal fusion directly at the membrane-potential level while inheriting the energy efficiency of spike-based computation, with roughly an order of magnitude lower inference energy than the similarly-sized ANN-based DCEvo.
Abstract
Infrared and visible image fusion (IVIF) integrates the complementary information of two modalities into a single image with richer scene content. While existing methods are largely built on artificial neural networks (ANNs), which densely compute over all activations, spiking neural networks (SNNs) communicate through sparse binary spikes and compute only where and when a spike occurs, offering a route to more energy-efficient fusion. However, directly applying SNNs to IVIF creates a fundamental tension: cross-modal fusion relies on fine-grained responses from both modalities, whereas binary spikes can discard complementary cues that remain below the firing threshold. The membrane potential retains these subthreshold responses before firing, letting both modalities jointly shape the output when integrated at this stage. Building on this, we propose CIS-Fuse, a spiking network that performs cross-modal fusion directly at the membrane-potential level. At its core is the current injection spiking (CIS) operator, which injects one modality as a gated auxiliary current into the driving neuron of the other, so the two integrate before spike firing, with a per-channel learnable injection strength that adaptively regulates the modulation magnitude. Building on CIS, we construct a bidirectional cross-modal fusion (BCMF) module and deploy it on a dual-branch architecture with asymmetric stacking depths, where the two branches develop a clear functional specialization. Extensive experiments on four IVIF benchmarks and on downstream detection and segmentation show that CIS-Fuse achieves fusion quality on par with state-of-the-art ANN-based methods while inheriting the energy efficiency of spike-based computation, with roughly an order of magnitude lower inference energy than the similarly-sized ANN-based DCEvo. Code will be released upon publication.
This work pioneers the synergistic integration of SNNs into Transformer architectures for LLIE, establishing a compelling pathway toward powerful, energy-efficient low-level vision on resource-constrained platforms.
Hongzhi Wang, Xiubo Liang, Jinxing Han et al.· Advances in Neural Informati...· 0 citations
Spiking Transformers model token interactions primarily through spiking self-attention (SSA). However, binary query and key representations map continuous similarities to sparse and discrete relation responses, which may suppress weak relations and limit the propagation of local spatial context. To address this limitation, we introduce Spiking Local Interaction (SLI) and Adaptive Complementary Fusion (ACF). SLI establishes an attention-independent pathway for direct information exchange among neighboring spiking tokens using lightweight depthwise--pointwise transformations. ACF integrates SSA and SLI through layer-specific, channel-wise coefficients that adaptively balance their contributions at different network depths. The proposed design preserves the original attention formulation and can be incorporated into different Spiking Transformer architectures with modest parameter overhead. Experiments on ImageNet-1K, CIFAR-10, CIFAR-100, CIFAR10-DVS, and ADE20K show consistent improvements across image classification, event-based recognition, and semantic segmentation. In particular, QKFormer with SLI and ACF achieves $84.37\%$ Top-1 accuracy on ImageNet-1K and $37.5\%$ mIoU on ADE20K, where the segmentation model is trained without ImageNet pretraining. Ablation studies and qualitative analyses further indicate that SSA and SLI capture complementary interaction patterns and that learnable fusion consistently outperforms fixed weighting.
Dongcheng Zhao, Sicheng Shen, Zhenyu Yang et al.· 0 citations
Semantic segmentation is critical for intelligent robotics to understand complex environments. While CNN-based models on RGB images achieve high performance, their accuracy drops in fast-motion or low-light scenes. Fortunately, event cameras, with high temporal resolution and low latency, offer robust perception in such challenging conditions. Many event-image fusion methods attempt to combine the complementary strengths of both modalities, but most adopt simple fusion strategies without considering intermodal correlations or designing computationally expensive architectures, resulting in degraded accuracy and high energy costs. To overcome these limitations, we propose a lightweight spiking neural network (SNN)-based event-image fusion network (Spike-EIFNet) that leverages the complementary strengths of multimodal fusion and energy-efficient spike-driven computation. In particular, to reduce computation cost for lightweight, Spike-EIFNet adopts a dual-branch SNN encoder to process events and images in parallel. Then, to improve the segmentation accuracy with enhanced feature interaction, we introduce a spike-driven cross-modal fusion (SCMF) module, consisting of a modality-aware fine-grained extraction (MFE) stage to capture dynamic cues from events and spatial details from images, followed by a cross-modal interaction and fusion (CIF) stage for effective feature alignment. Finally, a lightweight feature enhancement (LFE) module is proposed to further refine feature representations and facilitate deep-shallow feature fusion. Extensive experiments demonstrate that Spike-EIFNet achieves 67.34% and 58.09% mean intersection over union (mIoU) on the DDD17 and DSEC-Semantic datasets while consuming $72.83\times $ and $100.26\times $ less energy, respectively. Compared with ANN-based methods, Spike-EIFNet significantly reduces energy consumption; among SNN-based methods, it achieves the highest segmentation accuracy with a favorable accuracy-efficiency tradeoff. Code is available at: https://github.com/Chensyfighting/Spike-EIFNet.
Siyu Chen, Qie Liu, Xianlei Long et al.· IEEE Transactions on Neural...· 0 citations
Lapis is proposed, a spiking attention mechanism that scores each token pair by the L1 distance between its query and key first-spike latency vectors under time-to-first-spike coding, and maps this distance to an affinity through a Laplacian kernel.
Kaiwen Tang, Jiaqi Zheng, Zixuan Zhu et al.· 0 citations
Simulation results show that incorporating astrocytic modulation consistently enhances classification performance in leaky integrate-and-fire (LIF) networks, including under noisy conditions, and suggest that augmenting simplified astrocytic dynamics can improve robustness and computational capability in SNNs, while also increasing their biological plausibility.
D. Garcia, Sabir Jacquir· International Conference on...· 0 citations
SpikeRestormer is proposed, an energy-efficient SNN for AiOIR that performs event reasoning over internally generated spike cues that delivers competitive performance against ANN-based methods and establishes new state-of-the-art results among SNN-based methods with significantly lower energy consumption.
Shengkai Hu, Jie Shao, Jiaqi Ma et al.· 1 citation