Skip to content
Book Open access

LoopEdit: Learning Data-driven Hierarchical Planners for Short-Video Assembly

Nov 2026 · Proceedings of the 34th ACM International Conference on Multimedia · pp. 2721-2730 · 0 citations · 48 references

TL;DR

LoopEdit is presented, a hierarchical multi-LLM framework that converts a user directive and a set of input assets into an editable short-video plan and a rendered video and shows that data-driven hierarchical planning with preference alignment can produce more engaging and controllable short-form videos.

Abstract

Creating a compelling social media video from raw clips requires decisions at multiple levels, from story structure and pacing to clip selection and local editing effects. Existing language-based editing systems typically respond to explicit edit commands, retrieve clips, or generate videos directly, but they do not produce an inspectable plan that can be optimized for engagement. We present LoopEdit, a hierarchical multi-LLM framework that converts a user directive and a set of input assets into an editable short-video plan and a rendered video. LoopEdit first generates a high-level plan that defines the video arc, scene structure, pacing, and style, and then generates a low-level plan that maps scenes to specific clips, durations, effects, text, and narration. To train this system, we reformulate existing short-videos into asset-plan pairs through reverse-engineering. We also learn evaluator models from human annotations to predict how well a candidate video plan will perform, capturing both overall engagement and fine-grained attributes that drive it. These learned evaluators also serve as reward models to align the hierarchical planners with human preferences through reinforcement learning (GRPO). In our experiments, the learned evaluators correlate strongly with human judgments, and the optimized planners outperform strong zero-shot Multimodal LLM baseline in automatic evaluation and user studies. These results show that data-driven hierarchical planning with preference alignment can produce more engaging and controllable short-form videos.

Read PDF

Similar papers

#artificial intelligence Preprint Oct 2026

WeVisCut: The Whole Editing Room in One Agent From Narration to Long-Form Video

Producing long-form videos from narration requires interdependent decisions that remain coherent across shots. Existing agentic editors often use multi-stage workflows driven by general models, leaving the quality-cost trade-off of compact, task-adapted policies unclear. We address this gap with WeVisCut, a long-form s...

Chen-Yu Mu, Wen-Ming Weng, Qin-Rui Wang et al. · 1 citation
Preprint Nov 2026

Clips2Story: Training-Free Video Storyboarding and Editing Using Multimodal Retrieval-Embedded Generation

Video editors often work with large collections of fragmented footage and must select and organize relevant clips into a coherent story. However, many existing video editing and summarization methods assume a temporally ordered source video and aim to preserve its original narrative and temporal structure. In this pape...

Jun-Nuo Wang, Wei-Han Xu, Hao-Wen Dong · 0 citations
#artificial intelligence Preprint Oct 2026

ReVi: Recursive Recap Video Generation

Novels tell rich stories, but their textual form cannot directly present the sights and sounds of the worlds they describe. Film adaptations offer an audiovisual experience but require substantial production effort and viewing time; film recaps are shorter but depend on existing footage. Neither route directly provides...

Kyrie You Wu, Chao-Zhuo Li, Song-Yang Liu et al. · 0 citations
Preprint Oct 2026

Auto-Director: Leveraging Multi-Agent Collaboration for Autonomous Video Editing

Autonomous video editing from open-world resources remains a challenging problem in multimedia content creation. Retrieval-based methods are constrained by fixed source material and lack creative flexibility, while generative models struggle with long-horizon coherence and often diverge from user intent. Both paradigms...

Yu-Xiao Chen, Wen-Bo Zhu, Yang-Guang Ji et al. · 1 citation
Sep 2026

Complementary Retrieval-Augmented Prompting for Consistent Long-Form Video Generation

Quantitative and qualitative evaluations on multi-shot story generation demonstrate that the Complementary Retrieval-Augmented Prompting method consistently outperforms recent-frame, memory-based, and entity-level retrieval baselines in cross-shot consistency and text-controllability.

Xiang-Han Wei, Xiao-Da Yang, Zhi Wang et al. · 1 citation
Preprint Oct 2026

Hy-Nomas: Towards Free-form Interactive World Generation

Hy-Nomas is a video world model with an 8B diffusion backbone that supports free-form text instructions and continuous camera control across multiple interaction turns that generates variable-length segments conditioned on current inputs and cached history.

Qiao-Ling Chen, Jia-Xiang Cheng, Xue-Qing Deng et al. · 3 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.