Current status and future development of the key technologies for apple picking robots

Abstract: Fruit picking has been one of the most important operations in modern agriculture. Manual picking cannot fully meet large-scale apple production in recent years, due to the labor-intensive and time-consuming task. Alternatively, robot picking can be expected to replace manual work. However, the conventional semi-automatic picking machine can seriously vibrate the branches and damage the fresh apples during batch harvesting in the existing orchard. It is a high demand to detect apples under complex environments, and then pick them without damaging the fruit or trees. The picking robots are also required a compatible auxiliary platform during operations in an orchard. The performance of picking robots can be assessed using the picking speed, picking success, and fruit damage rates. These parameters depend on three key technologies: target recognition, localization, and fruit separation. In this paper, an overview was provided of the history and current status of commercial apple-picking robots. Specifically, the performance and robustness were highlighted, in terms of the picking efficiency, structural composition, the type of orchard, and adaptability to the external environment. The main technical features were then summarized from the application of apple-picking robots. Since the apple picking was operated in an unstructured orchard environment, the most serious challenge remained in the application of target recognition, localization, and fruit separation techniques. Therefore, the recognition techniques were reviewed for the apples in the complex natural environments with changing light, branch and leaf shading, overlapping fruit, night-time environments, and same-colored apples. Deep learning-based apple recognition algorithms were the far-reaching prospects in complex environments, particularly in multiple scenarios. The accuracy and speed of apple recognition were improved using deep learning recognition algorithms in the field of apple picking. The second most challenge was the application of the apple localization for the fruit occlusion and overlap, while the fruit displacement by wind during the imaging process. Therefore, four aspects were introduced in the apple localization, including 2D information acquisition, depth information acquisition, apple pose acquisition and oscillating fruit localization. Point cloud processing algorithms were highlighted to achieve the precise localization of occluded apples. In addition, the applications of end-effectors were reviewed in the apple-picking robots with high efficiency and non-destructive picking. Three types of the end-effector structure (such as hand-jaw, vacuum, and shear) were summarized under fruit separation. It was found that the hand-jaw was the most commonly-used type of end-effector. Current research on the end-effectors was focused on the flexible materials for the finger surfaces, under-driven, flexible structures or control systems, such as the soft hands and perception-based servo control. Therefore, it is necessary to improve the picking speed and success rate with the low fruit damage and cost saving in the apple picking robot, in order to accelerate the process of apple picking automation. Future development was also proposed for the apple-picking robot in the urgent commercial need of the pioneering innovation in the combination of agricultural machinery and agronomy. Some optimization can be expected to be performed on the recognition algorithm, multi-sensor fusion, multi-arm cooperation, human-machine collaboration, extending device versatility, and integrating 5G and IoT technology. This finding can provide a strong reference for the current status, future development trends, and equipment innovation of apple-picking robots, together with the key technologies.