Learning Subject Tags from Incomplete Labels: A Recall-Oriented VLM Pipeline for Historical Photographs

Abstract

Assigning subject tags to photographs is essential for access and discovery in digital libraries, yet doing it by hand does not scale. Vision–language models (VLMs) are a powerful and increasingly popular tool for such tasks, but prompting them to generate open-vocabulary subject tags directly yields unsatisfactory results. Fine-tuning is the natural remedy, and here a second challenge arises: human-curated tags are an imperfect supervision signal: many true tags are missing, and the recorded ones appear in an arbitrary order. To address these challenges and improve the recall of meaningful tags, we propose a VLM fine-tuning pipeline for subject tagging of U.S. historical photographs. The pipeline begins with LoRA supervised fine-tuning. To counter the arbitrary tag order, we introduce an order-invariant training scheme that shuffles the target tags per example. To further raise recall, we continue training with reinforcement learning, specifically Group Relative Policy Optimization (GRPO), under a recall-oriented reward. Finally, we apply a test-time scaling technique, self-consistency, to push performance further at inference. Evaluated on a ground truth rebuilt from the Digital Public Library of America, the combined techniques improve Recall@10 by + 0.51 (a ∼ 10 × gain) over the off-the-shelf VLM and by + 0.13 (\(+29\%\) relative) over plain LoRA fine-tuning.