From Vision to Grasp: TCP Dynamic Realignment Using Embedded Sensors

Abstract

The integration of robotics into agriculture addresses challenges such as labour shortages and sustainable production. Handling delicate and irregular natural products remains difficult for conventional grippers. This paper presents a baromorphic end-effector as a proof-of-concept platform for validating a perception-sensing TCP realignment pipeline rather than as a field-ready agricultural product. The square-to-hexagonal layout choice was made iteratively through prototyping, whereas a finite-element analysis was used in prior work to optimise and characterise the hexagonal cushion unit. The materials, actuation, and robotic platforms serve as the experimental testbeds for the integrated control concept. The system uses a modular array of pneumatically actuated hexagonal cells with sparse embedded pressure sensing, vision-based object detection, a discrete heuristic visual servoing controller to bypass kinematic singularities during the approach, and a closed-loop TCP dynamic realignment algorithm. Under controlled laboratory conditions, the prototype achieved 12/12 successful linear grasps and 9/11 successful angular grasps (82%) on irregular produce, whereas a conventional rigid parallel-jaw gripper failed all attempted baseline trials under the same protocol.