DGS-Net: Learning Dexterous Grasps From Single-View Point Clouds via a Multi-Object Scene Dataset

Abstract

Dexterous grasping in multi-object scenes remains a fundamental challenge for robotic manipulation, particularly under single-view perception where occlusion and incomplete geometry introduce significant uncertainty. To address this challenge, we propose DGS-Net, an end-to-end dexterous grasp prediction network that directly infers complete high-degree-of-freedom grasp configurations from single-view scene point clouds without relying on explicit object reconstruction or segmentation. By explicitly modeling grasp offsets and pre-grasp configurations, DGS-Net effectively compensates for projection errors and geometric ambiguity caused by partial observations, enabling more accurate and physically consistent grasp execution. To support effective training of the proposed model, we develop an efficient two-stage dexterous grasp data generation pipeline that first synthesizes dense object-level grasps and then refines them into physically consistent scene-level grasps. Our dataset provides explicit grasp offset modeling, pre-grasp configurations, and point-cloud-level grasp semantic labels, which guide the network to focus on grasp-relevant regions, thereby enhancing the precision and robustness of grasp prediction under occluded conditions. Extensive real-world experiments demonstrate that DGS-Net achieves a grasp success rate of 79.39% and a clearance rate of 87.94% on a physical robotic platform. Moreover, compared to the baselines, our method reduces the translation error by approximately 29% and the rotation error by about 41% in grasp perturbation tests, validating its effectiveness, robustness, and generalization performance for single-view dexterous grasping in multi-object scenes. Our dataset is available at https://github.com/4taotao8/DGS-Net Note to Practitioners—Dexterous grasping is a fundamental capability for robots, and this work aims to address the practical challenges of deploying multi-fingered dexterous hands in unstructured industrial environments such as intelligent manufacturing and automated sorting. Conventional grasping systems often perform poorly in multi-object scenes because a single camera cannot observe object regions that are occluded. To address this limitation, we develop DGS-Net, an end-to-end framework that directly predicts dexterous grasp configurations from single-view scene point clouds without requiring explicit object segmentation or shape reconstruction. To improve grasp reliability under partial observations, the proposed method explicitly models grasp offsets and pre-grasp configurations. These mechanisms help compensate for perception errors caused by occlusions and sensor noise, enabling more accurate dexterous grasping and safer approach motions. In addition, we introduce a cost-effective two-stage data generation pipeline that produces dexterous grasp annotations with semantic labels and grasp offsets, achieving comparable grasping performance with a much smaller dataset, thereby significantly reducing the cost of dataset construction for practical deployment. The current limitation is that the method mainly focuses on the hand itself and is, to some extent, decoupled from the kinematic constraints of the robotic arm. Future work will integrate arm motion planning with grasp prediction, incorporate multimodal sensing such as tactile feedback to adapt grasps to objects’ physical properties, and further enrich the diversity of dexterous grasp patterns.