Sonia Mazelet

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

This work introduces a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data and proposes SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher...