AI-Assisted Metadata Enhancement for Digital Image Collections: A Pilot Study of Vision-Enabled Language Models in Cultural Heritage Description

Abstract

This study evaluates the efficacy of Large Vision-Language Models (LVLMs) for remediating and creating metadata within the Pittsburgh City Photographer Collection. By comparing outputs from several OpenAI and Google Gemini models against a human-remediated baseline, the research identifies a significant performance gap between AI-assisted remediation and autonomous metadata creation. Findings demonstrate that while LLMs can help institutions “see” obvious visual features previously omitted from legacy records, they lack the specialized local knowledge required for authoritative description. Subject heading application failed to align with established taxonomies, and models frequently introduced inappropriate editorial “noise” in descriptions. Most concerning were the error rates: 20% of objects contained factual errors in titles or descriptions when baseline metadata was provided, jumping to 40% when no context was available. These errors represent a direct threat to archival integrity. The study concludes that the expertise required to detect these sophisticated hallucinations often exceeds the effort of original metadata creation. Consequently, the researchers determined that the risk of introducing errors outweighs the benefit of increased data volume. Rather than moving forward with AI for a 20,000-image backlog, the institution will focus on updating human-centered descriptive practices and documentation to ensure metadata accuracy and reliability.