It is shown that self-conditioning corrects a train/inference mismatch by exposing the model to inference-like states that are otherwise rare during training, and cross-entropy drives embeddings towards a geometry whose decoding error rate closely matches the one-hot one.
Oscar Davis, D. Roos, Alice Bizeul et al.· 0 citations
PUMBA is introduced, a unified framework for trajectory-aware training that trains the denoiser on consecutive steps of policy-induced trajectories, passes information between steps, and optimizes them jointly by backpropagation through time.
Manuel Madeira, Amitis Shidani, Alice Bizeul et al.· arXiv.org· 2 citations· ⚡1
It is shown that a step matches the training distribution only when the positions it writes are conditionally independent given the tokens already fixed, that no product of per-position distributions can match a dependent group, and that per-position distributions do not determine whether a group is dependent.
Russ Webb, Amitis Shidani, Alice Bizeul et al.· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.