Design and test of a four-arm apple harvesting robot

A four-arm harvesting robot system was designed to integrate with the fruit picking-collecting-transporting multifunction for the apples’ automatic harvesting. Taking the standardized tall-spindle and dwarf-rootstock apple tree as the object, the target operational area was determined for the harvesting robot, according to the fruits’ spatial distribution within the tree canopy. A new configuration of a four-arm picking manipulator and the operational mode were proposed with the four Cartesian coordinate arms in the three degree-of-freedom (DOF). An electric-pneumatic hybrid dual-stage driving structure was utilized to ensure efficient and large-scale telescopic motion within the tree canopy. Additionally, a CAN open bus-based integrated drive-control harvesting gripper was designed to enable efficient harvesting operations via a combination of fruit gripping and twisting actions. A multi-task deep convolutional network was adopted to recognize the fruit’s discrete visual pixel areas that were caused by branches and leaves occlusion. As such, the semantic segmentation of the occluded fruits and end-to-end determination of the discrete areas’ ownership were realized to overcome the traditional single-task networks in the classification of discrete regions of the same fruit. The view frustum projection model was introduced to locate the centroid of the target fruit, according to the local point cloud information on the surface. A novel strategy of four-arm picking task area partitioning was proposed, according to the clustered distribution characteristics of fruits within the tree canopy. The time-optimal four-arm collaborative picking task planning was also proposed to achieve the efficient traversal of different regions inside the tree canopy by the robotic arms. Finally, the key components of the harvesting robot were integrated to develop the autonomous harvesting workflow. The production trials were also conducted in a high-density dwarf rootstock orchard. The results showed that the recognition rate on the visible fruits was 92.94%, among which 90.27% of the fruits’ positioning accuracy was sufficient for picking operations. The robot’s average overall picking efficiency was 7.12 seconds per fruit, among which the efficiencies of single-, dual-, and four-arm were 9.59, 8.17, and 4.87 seconds per fruit, respectively. The efficiency of four-arm collaborative picking was approximately 1.96 times that of single-arm picking. The success harvesting rate of visible fruits was 82.00%, and the overall harvesting rate for all fruits inside the tree canopy was 74.56%. The success rate of harvesting reached up to 100% in the outer peripheral areas of the tree canopy where the fruits were sparse. However, the success rates of target recognition, location, and operation were significantly lower in the inner-dense region where the fruits were intensive, resulting in a harvesting success rate of 73.63%. The harvesting failures were attributed to the fruits that were obstructed by branches and leaves, leading to the visual recognition and positioning accuracy, as well as the interference and collisions with the harvesting manipulator. Therefore, the robot's capability of autonomous obstacle avoidance was enhanced to improve the tree structure and the performance of this harvesting robot. This finding can be considered as the preliminary exploration for the development and application of robotic harvesting models for freshly-eaten fruits.