The key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling and replacing global cross-attention with AVS-Conv, a geometry-aware local aggregation operator operating within local neighborhoods rather than the exhaustive latent set.
Abstract
High-fidelity 3D generative modeling increasingly relies on the latent diffusion paradigm, where the reconstruction quality of the underlying 3D VAE becomes a primary bottleneck. Existing approaches largely follow two paradigms: sparse voxel-based representations achieve strong reconstruction quality but incur significant memory and computational overhead, while set-based representations are compact and continuous yet typically lag in fidelity due to latent sparsity and excessive global smoothness. We propose MSVS-VAE, a hierarchical set-based VAE that closes this fidelity gap without sacrificing compactness. Our key idea is to progressively densify anchored VecSet latents via hierarchical point-shuffle upsampling, increasing spatial capacity for fine-grained geometry modeling. To efficiently decode from the densified hierarchy, we replace global cross-attention with AVS-Conv, a geometry-aware local aggregation operator operating within local neighborhoods rather than the exhaustive latent set. We further introduce multi-scale query decoding to fuse coarse-to-fine latent features, where coarse scales provide stable global context, and fine scales refine localized geometry, reducing artifacts from overly local receptive fields. Extensive experiments on Objaverse, ABO, and in-the-wild benchmarks demonstrate that MSVS-VAE consistently outperforms prior set-based and voxel-based VAEs, delivering approximately 10x faster decoding than prior set-based methods and approximately 10x higher compactness than voxel-based baselines.
Sparse voxel grids preserve the spatial structure needed for detailed 3D reconstruction, but their memory still grows rapidly with resolution as active surface cells increase. We introduce ChunkVAE, a sparse grid variational autoencoder organized around local chunks rather than a global latent volume. Local learned operators permit independently chosen encoder and decoder partitions and allow inference chunk sizes to differ from training. Two complementary data operators make this flexibility practical: Balanced Binary Object Partitioning distributes active cells while limiting replicated overlap, while S-Curve weighted stitching attenuates unreliable boundary features when assembling a global latent or reconstruction. Across three object benchmarks, ChunkVAE is competitive with or better than strong baselines from $512^3$ to $1536^3$; smaller chunks lower peak allocated memory and shorten per-chunk compute, enabling faster parallel inference. Stable stitched latents and improved image to 3D metrics indicate that local compression can scale geometry while retaining the global interface required downstream.
Kaiyi Zhang, Zhihao Liang, Haolin Liu et al.· 0 citations
Large-scale 4D scene generation aims to synthesize dynamic 3D environments and provides a critical intermediate representation for downstream tasks such as autonomous driving simulation, embodied agent training, and scene forecasting. Existing methods typically adopt a two-stage latent diffusion paradigm, which improves computational efficiency by modeling and generating scenes in a compressed latent space. However, this paradigm suffers from a reconstruction-generation trade-off: increasing the latent dimensionality improves reconstruction fidelity, but substantially increases the computational burden of diffusion modeling and makes generative optimization more challenging. This issue becomes particularly pronounced in 4D occupancy generation, where complex spatial layouts and long-range temporal dynamics must be jointly preserved within compact representations. To alleviate this problem, we advance a central principle: low-dimensional latent spaces should not rely solely on unconstrained compression, but should instead be structurally regularized to preserve sufficient 4D spatio-temporal information while maintaining compactness. To instantiate this principle, we propose Compact Autoregressive Latent Prior (CALP), which regularizes low-dimensional latent variables with a history-conditioned autoregressive prior, achieving compact, long-range temporally coherent, and spatially structured latent representations through a more reasonable latent capacity allocation and a latent-space organization that is better suited for diffusion-based 4D generation. Built upon CALP, we further introduce 4DWeaver, a compact 4D scene generation framework that enables high-quality spatio-temporal occupancy synthesis in a low-dimensional latent space. Extensive experiments on multiple large-scale 4D occupancy benchmarks demonstrate that 4DWeaver achieves superior reconstruction and generation performance while substantially reducing memory consumption and computational cost.
Ruosen Hao, Tianyu Shen, Jing-Yuan Gao et al.· IEEE Transactions on Pattern...· 0 citations
This work introduces VAR-VTON, a VAR-based VTON model that incorporates garment conditioning and structural guidance for efficient latent-space VTON, and proposes STAR-VTON, a Two-Stage AutoRegressive framework that builds upon VAR-VTON by decoupling latent-space structural synthesis from pixel-space detail recovery.
Despite recent advancements in native latent diffusion models for single-view 3D generation, they still suffer from inadequate geometric details due to the global entanglement of VecSet-based representations and inaccurate structural topology arising from single-view ambiguity. Existing solutions often rely on model scaling over large-scale datasets to improve fidelity, which is computationally prohibitive and inefficient in capturing fine-grained details. In this work, we propose Tempo3D, a resource-efficient paradigm for high-fidelity VecSet-based 3D generation via efficient temporal-aware fine-tuning and multi-view latent aggregation. Our approach consists of two critical components: (1) To facilitate robust detail learning, we establish the TempoDetail dataset and design the Time-Segmented LoRA (TS-LoRA) module. TS-LoRA exploits the inherent "structure-to-details" temporal transition of flow-based models to decouple feature learning, significantly bolstering the capacity to produce high-frequency details without large-scale retraining. (2) To mitigate geometric inaccuracies stemming from single-view ambiguity, we propose a multi-view latent aggregation strategy incorporating Soft Flow Trajectory Projection (Soft FTP) and Score-based Velocity Field Aggregation (SVFA). By aggregating reference views derived from Multimodal Large Language Models (MLLMs), these components resolve the intrinsic conflicts between velocity vectors caused by cross-view latent discrepancies, thereby preventing geometric collapse and ensuring directional consistency. Moreover, the integration of MLLMs empowers Tempo3D to support precise semantic 3D editing via natural language prompts while maintaining structural integrity. Experiments demonstrate that Tempo3D outperforms state-of-the-art baselines in generation quality, geometric precision, and editability under a highly efficient computational budget.
Huizhi Zhu, Jiongming Qin, Yusen Wang et al.· ACM Transactions on Graphics· 0 citations
MotionCraft is presented, a controllable VSR framework that formulates restoration as motion-aware latent state prediction inspired by world models and integrates adaptive sparse attention with an explicit user-accessible control interface to deliver temporally consistent, high-quality reconstructions under streaming constraints.
Rong Fu, Chunlei Meng, Yangcheng Zeng et al.· 0 citations