CWAM (Causal WAM), which incorporates the sparsity constraint into a pretrained WAM by adding a sparse routing path to its action head, with the world modeling objective remaining unchanged, is proposed.
Jun-Tao Cheng, Jin-Zhou Tang, Fan Feng et al.· 0 citations
SplitJEPA is introduced, a JEPA that jointly recovers the latent state and its invariant and variant organization directly in representation space, without any reconstruction, and extends reconstruction-free latent recovery to invariant-variant block identification.
Rui-Jin Hua, Zichuan Liu, Zhuokai Zhao et al.· 0 citations
It is shown that small models can allocate learning capacity inefficiently toward early distributions, whereas sufficiently overparameterized models are robust to this effect, and Exposure Therapy (ET), a simple regularization that promotes more efficient allocation of learning capacity during sequential pretraining, i...
DSReg (Dependency-Sparsity Regularization) recovers individual world latents up to signed permutation, without reconstruction or a decoder, without reconstruction or a decoder, and establishes the first fully identifiable JEPA that recovers every world latent.
Yujia Zheng, David A. Klindt, Randall Balestriero et al.· 0 citations
This work characterize structural misspecification as a perturbed factor analysis problem, and establishes subspace identifiability under spectral separation and controlled perturbation, followed by component-wise identifiability under structural sparsity.
Yu-Ke Li, Yujia Zheng, Zi-Yi Chen et al.· 0 citations
A condition called the cross-Hessian Rank Constraint (HRC), which serves as a primitive rank-based tool for nonlinear latent causal discovery, shows that a rank-based property arises from the cross-Hessian of the observed-data log-density in the nonlinear case, revealing information about the latent variables.
Zi-Jian Li, Rui-Chu Cai, Feng Xie et al.· 0 citations
Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicit...
Shikai Qiu, Marc Finzi, Yujia Zheng et al.· arXiv.org· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.