Hierarchical Token Learning with Discriminative Training for Generative Text-to-Image Retrieval

Abstract

Generative retrieval reformulates text-to-image retrieval as autoregressive generation of discrete image tokens, offering a promising alternative to conventional dense-retrieval paradigms. However, existing approaches typically quantize a single global visual representation, struggling to capture the hierarchical semantics of images spanning scene-level context and entity-level details. Furthermore, the standard training objective maximizes only target token likelihood, lacking discriminative ability against semantically similar but irrelevant candidates. To address these, we propose HiToken, which decomposes visual semantics into scene-level and entity-level representations quantized by separate residual codebooks for structured hierarchical tokens, and leverages an LLM to decompose captions into scene and entity descriptions for level-specific cross-modal supervision. A dual hard negative discriminative training strategy further mines image-side negatives via token prefix overlap and text-side negatives via semantic similarity, jointly optimized with the generation objective through ranking losses to enhance discriminative power. Extensive experiments on MS-COCO and Flickr30K demonstrate that HiToken outperforms existing generative retrieval methods. The code is available at https://github.com/immc-lab/HiToken.