From Sketch to Scene: Relightable Garment Reconstruction from Marked Sketches
Abstract
Sketching is one of the most natural ways for people to express visual ideas. Unlike the text prompts that most generative systems rely on today, a sketch captures spatial structure and intent directly, including where a designer wants highlights and shadows to appear. Yet current sketch-to-3D pipelines have no input for that intent: they bake the drawn shading into the texture as dark paint, and the light cannot be moved. We propose a system that transforms a rough garment sketch, marked with sparse annotations (a star for a highlight and a moon for a shadow), and a short text prompt describing the texture, into a textured 3D mesh with an explicit directional light, as a scene that can be relit and edited after generation. The two symbols are the most frequent proposals in an elicitation study with art students. From the marked sketch the system predicts the scene's directional light and, separately, a displacement field that bends the surface so that the moon-marked regions fall into shadow under that light, rather than being painted dark in the texture.
