TL;DR
The experimental results show that even 130M-parameter models benefit from including the MTP task in the pre-training objective, and hold even under severe data constraints, as demonstrated on both zero-shot benchmarks and downstream tasks.
This work finds that likelihood-trained TPMs can result in failed generations due to overly large corrections to the LM’s logits, and trains TPMs with LM-aligned objectives that better align with the LM token-probability space.
Hierarchical Latent Prediction (HiLP) is introduced, which introduces an auxiliary higher-level abstract latent to help reduce the error accumulation effect in latent-space rollouts and can lead to longer-horizon coherent belief state representation.
Pretrained language models (PLMs) have established state-of-the-art performance across diverse natural language understanding (NLU) tasks. This study reveals that seman-tic-rich explanations of lexical units can effectively guide PLM learning processes. We propose a novel language understanding enhancement method with...
The results suggest that competitive MLLM can emerge from alignment alone, reducing multimodal extension to a lightweight projector-training problem that generalizes across modalities and adapts rapidly to each new LLM release.
A simple method, Self-Guided TTT (S-TTT), which improves accuracy for both Qwen3-4B-Thinking-2507 and Llama-3.1-8B-Instruct, achieving up to a 15% relative improvement.
AdaMTP is proposed, an adaptive training paradigm that dynamically aligns the prediction horizon with the intrinsic predictability of the sequence, and consistently outperforms standard MTP in both task performance and inference speedup.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
