Making Language the Low-Rate Code of Vision: Successive Refinement from Words to Pixels
An ordered visual code lets a short prefix stand in for an image, and the training loss decides what that prefix contains. Under pixel reconstruction the first bits go to the directions of highest variance, and we prove that a caption can then be independent of the entire first half of a code that is rate-optimal for p...
Rui-Lin Gong, Hao Gong
· 0 citations