Instructable 3D Scene Editing via LLM-Driven Closed-Loop Gaussian Splatting

Abstract

Creating traditional 3D scenes is both time‐consuming and costly, requiring designers to meticulously configure 3D assets and environments. Recent advancements in generative AI, including text‐to‐3D and image‐to‐3D methods, have significantly reduced the complexity and cost of this process. However, current techniques for editing complex 3D scenes still rely heavily on interactive, multi‐step 2D‐to‐3D projection methods and diffusion techniques, which often lack precision. Therefore, this study proposes AgentEditor —— a controllable 3D scene editing and multimodal interaction framework that integrates 3D Gaussian splatting with large language models (LLMs). It establishes an LLM‐driven “edit‐evaluate‐optimize” autonomous closed‐loop feedback mechanism capable of quantitatively assessing semantic degradation after editing and dynamically triggering adaptive local semantic fine‐tuning. Extensive experimental results demonstrate that AgentEditor achieves superior editing accuracy and speed compared to current state‐of‐the‐art 3D scene editing methods, setting a new benchmark for efficient interactive 3D scene customization.