Research advance on vision system of apple picking robot

Vision system is one of the most important parts of apple picking robot, which, to some extent, determines the quality and the speed of picking task implemented by apple picking robot. In this review, we enumerated the existing apple picking robots. Some information, such as type of visual sensors, hardware of vision system, success rate of harvesting and run time, was illustrated in details. Meanwhile, on the basis of discussing the vision system of apple picking robot, we focused on summarizing hardware structure of existing vision systems and apple image segmentation methods as well as apple recognition and localization methods applied in vision systems. The vision system of apple picking robot mainly includes machine vision system, laser vision system, three-dimensional vision system and vision system formed by machine vision and other vision system. And the machine vision system can be classified into 3 types according to the number of image sensors used, that is, monocular vision system, binocular vision system and multi vision system. The recognition and localization of apple target is the first step of the implementation of the picking task for picking robot. The currently used apple segmentation and recognition methods include threshold segmentation algorithm, chromatic aberration based algorithm, K-mean clustering algorithm, region growing algorithm, segmentation method combining 2 or more algorithms, and so on. There are 4 methods that are commonly used in the localization of apple target. They are the methods based on centroid, fitting circle, symmetry axes, and three-dimensional coordinates, respectively. In natural scene, the recognition and localization of apple target may be affected by many factors. Hence, recognition and localization of apple target under different conditions, such as color nonuniformity, different illumination, shadow on the surface, oscillation, overlapping and occlusion, was reviewed and analyzed. Among all these conditions, occlusion can be regarded as the most serious factor. The condition of occlusion can be roughly divided into 4 kinds, i.e. apple target blocked by other apple, by branches, by leaves and by branches, leaves and other apple simultaneously. As for apple targets blocked by branches, one apple may be separated by branches, thus causing that an apple may be recognized as several apples. For the apple targets blocked by leaves, the symmetry of apple can be utilized to localize apple targets. The apple targets blocked by other apple can be considered as overlapping. There are overlapped apples with series connection, parallel connection, and blend connection. Because of the complexity of overlapping, the recognition and localization of apple targets blocked by other apple target is a little more difficult. In addition, the detection of obstacles like tree trunk and branches in apple orchard is important for apple picking robots to avoid obstacles, and thus obstacles detection methods were summarized in this review. In the process of target recognition and localization, binocular vision technology was commonly used in vision system. The key point of binocular vision technology is stereo matching. Therefore, stereo matching was then reviewed, and the image matching methods in existence can generally be divided into 2 categories, i.e. region-based image matching method and feature-based image matching method. What's more, the problems exist in recognition and localization methods used in the vision system of apple picking robot, including accuracy, effectiveness, character of real-time and universal applicability, were analyzed. Further study will concentrate on optimizing the structure of vision system, optimizing the intelligent algorithms used in the vision system, improving the real-time capability, recognizing and locating apple targets when the apple targets and vision systems are influenced by oscillation, and improving cost performance. The paper has summarized and analyzed vision system of apple picking robot comprehensively, which can provide reference for future research.