Skip to content
Open access

CLS-Guided Multimodal Fusion for Image Captioning with LSTM and T5 Decoders

Oct 2026 · Applied Sciences · 0 citations

Abstract

Turning visual evidence into coherent sentences depends on how visual and linguistic representations are combined before decoding. We present two decoder-specific generation strategies, CLS-LSTM and CLS-T5, built on a common CLS-guided Swin–BERT multimodal backbone. Swin-Base extracts a 7×7 grid of 49 spatial feature vectors, each with 1024 dimensions, which are mapped to 512 dimensions through a trainable linear projection initialized from principal components fitted exclusively on the training split. Under a leave-one-out training strategy, one caption serves as the generation target, while the remaining four captions are concatenated and encoded by BERT. The final-layer BERT [CLS] representation is used as the query in multihead cross-attention over the projected visual features, producing a global multimodal conditioning representation. Batch-level modality dropout exposes both decoder configurations to the image-only condition used during validation and testing. CLS-LSTM uses the fused representation for global conditioning, while the previous hidden state of its first LSTM layer queries coverage-aware Bahdanau attention at each decoding step. CLS-T5 projects the fused representation to the visual feature dimension, injects it into the projected visual feature sequence, refines the conditioned sequence using a single Transformer encoder layer, and supplies the resulting visual memory directly to the T5-small decoder while bypassing the original T5 encoder. When auxiliary text is unavailable, CLS-LSTM replaces the BERT [CLS] query with a learned null-text query, whereas CLS-T5 bypasses the auxiliary text fusion branch and conditions the visual features using a learned fusion bias. We evaluate both decoder configurations on the Karpathy splits of Flickr30K and MS COCO 2014 under image-only inference. On Flickr30K, across three random seeds, CLS-LSTM achieves mean BLEU-4 and CIDEr scores of 23.97±1.05 and 50.92±1.08, respectively, whereas CLS-T5 achieves 27.00±1.09 and 64.84±2.86. On MS COCO 2014 for seed 0, CLS-LSTM obtains BLEU-4 and CIDEr scores of 32.16 and 105.18, while CLS-T5 obtains 35.38 and 117.15. These results show that the shared CLS-guided framework supports both recurrent and Transformer-based caption generation while preserving image-only inference.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.