EADIC: Entity Aware and Diverse Image Captioning Utilizing a Fusion Transformer

Abstract

Image captioning is a significant vision language task that supports accessibility, human-AI interaction, and image retrieval. Despite the field's continuous progress, the generated captions remain generic, repetitive, and uninformative. To address these issues, this study proposes Entity Aware and Diverse Image Captioning (EADIC), a multimodal image captioning model. The proposed model generates accurate, diverse, and entity-aware image descriptions by incorporating named entities such as persons, locations, and organizations whenever sufficient context is available. The proposed framework includes three information streams: visual feature tokens from a caption-oriented novel Convolutional Neural Network (CNN) encoder, object tokens, and named entity information tokens, fused using a fusion transformer, which utilizes multi-head attention. The caption-generating transformer decoder is first trained employing cross-entropy loss, which is later fine-tuned using reinforcement learning to maximize accuracy, entity coverage, and diversity. Experiments on benchmark datasets demonstrate that the proposed EADIC model outperforms state-of-the-art captioning methods, achieving an Entity F1-score (EF1) of 31.1 for entity-aware captions and CIDErBtw of 52 for diversity, without compromising fluency and accuracy as reflected by a CIDEr score of 71.