Construction of a fine-grained retrieval model for archival text-image based on the integration of scene graph generation and attention mechanism

Abstract

To tackle the challenges in fine-grained retrieval stemming from noise, official seal occlusions, small text blocks, and other issues prevalent in archival text images, and to fulfill the requirements of integrating both textual and visual dual features while enhancing retrieval accuracy and efficiency, this study has devised a five-tier architectural model. This model comprises an input layer, a preprocessing layer, a scene graph generation layer, an attention fusion layer, and a retrieval matching layer. The model incorporates a dedicated scene graph generation module tailored for archival data, aiming to enhance element detection. Additionally, it features a three-tier attention fusion module that integrates scene graph, text, and cross-modal features to ensure precise feature alignment. Training is carried out using a multi-task loss function, and an index is created to streamline retrieval and matching processes. Experimental results show that the proposed model achieves a Top-1 accuracy of 83.7% and an average precision of 88.3% on the test set, representing a 25.1% improvement over the Top-1 accuracy of an optical character recognition (OCR) combined with word frequency and inverse document frequency model. The proposed model achieves a Top-1 accuracy of 6.1% higher than the archival retrieval network model for examples with official seal occlusion and a Top-1 accuracy of 76.8% for small text blocks. The response time for a single retrieval is 52.6ms. Research provides technical support for efficient retrieval of large-scale archives in archives, effectively solving the problem of archive retrieval in complex scenarios, and significantly improving the efficiency of archive management and utilization.