Abstract:
Abstract: Spatial position and coordinate points (called picking points) can widely be visualized in intelligent robots for fruit picking in mechanized modern agriculture. Recognition and location of picking points have also been the key technologies to guarantee the efficient, timely, and lossless picking during fruit harvesting. A tomato cluster can be both mature and immature tomato fruits, particularly in various shapes. Meanwhile, the color of fruit stem is similar to that of branches and leaves, while, the shape of fruit stems and petioles are similar. As such, there are large depth value errors or even a lack of depth values captured by the economical RGB-D depth camera using active stereo technology. Therefore, it is very difficult for picking robots to identify the picking points of tomato clusters in a complex planting environment. In this study, a recognition and location algorithm was proposed for the picking points of tomato clusters using RGB-D information fusion and target detection. Firstly, the Region of Interest (ROIs) of tomato clusters and stems were collected via the YOLOv4 target detection, in order to efficiently locate picking targets. Then, the ROIs of pickable stems that connected to the ripe tomato cluster were determined by screening, according to the neighbor relationship between the tomato clusters and stems. Secondly, the comprehensive segmentation was selected using RGB-D information fusion, thereby to accurately recognize the picking points of stems against the ROI color background. Specifically, the tomato clusters from the nearest row were regarded as the foreground in the RGB-D image, while the rest were assumed as the background (i.e., noise), due mainly to only that the nearest row for picking in robots. After that, the depth information segmentation and morphological operations were combined to remove the noise in the pickable stem ROI of RGB images. Subsequently, the pickable stem edges were extracted from the stem ROI using K-means clustering, together with morphological operation and RGB color features. The center point of skeleton along the X axis was set as the picking point (x, y) in image coordinate system, especially after extracting the skeleton of stem via the thinning operation. Thirdly, the RGB image and depth map of pickable stem ROI were fused to locate the picking point. Specifically, the average depth of pickable stem was calculated using the depth information of the whole pickable stem without the noise under the mean filter. Correspondingly, an accurate depth value of picking point was obtained to compare the average with the original. Finally, the picking point was converted to the robot coordinate system from image one. Eventually, the harvesting robot implemented the picking action, according to the coordinates of picking point. A field test was also conducted to verify, where the average runtime of one image was 54 ms, while the picture resolution was 1 280×720, the recognition rate of picking points was 93.83%, and the depth value error of picking point was ±3 mm. Thus, the proposed algorithm can fully meet the practical requirements during field operation in harvesting robots.