An apple picking point localization method based on semantic segmentation of target region

Apples are rich in vitamins and fibre, which help to protect the health of the body, and are popular in daily life. Traditional apple picking requires a large amount of human input and is inefficient, so how to automate apple picking is also a current research hotspot. To automate apple picking, one of the key issues is to accurately locate ripe apples. Traditional methods usually rely on manual labelling or rule-based computer vision algorithms to identify fruit locations, but in complex and changing orchard environments, the intertwining of branches and leaves of fruit trees, the shading of fruits, and the changing light conditions make the traditional counting and positioning methods unable to meet the needs of complex environments and different seasons. Therefore, it is necessary to propose accurate and efficient algorithms to achieve apple picking point localization. To address the above issues, we propose an efficient positioning method for apple picking points based on target area segmentation in this study. The specific process is as follows. Firstly, the acquired target detection frame is cropped to obtain the apple target region, and the LabelMe annotation tool is used to manually annotate the outer contour of the target point by point to construct the semantic segmentation dataset of the apple target region. A total of 1503 images were obtained after the above operations, of which 1352 were used for training and the remaining 151 were used for validation. Secondly, MobileViT-Seg, an apple target region semantic segmentation algorithm, is proposed to construct a lightweight encoder and a hierarchical pooling decoder. The encoder part adopts the pre-trained MobileViT structure, which downsamples the input images step by step to extract high-level feature information. The decoder part, on the other hand, uses the PPM (Pyramid Pooling Module) module and Softmax processing to gradually recover the spatial resolution of the image and generate accurate segmentation results. Effective extraction of global contextual information is maintained while keeping a small model size and low computational cost. Finally, to address the problem of incomplete apple region due to branch and leaf occlusion and fruit overlap, the target mask region obtained by segmentation is fitted with a circle using the least squares shape fitting method, and the center of the circle is used as the location of the picking point, and the RGB-D information is fused to achieve the localization of the spatial location of the picking point. The experimental results show that the proposed MobileViT-Seg model has high robustness in locating the picking point in multiple scenes. Comparing the proposed model with several mainstream segmentation algorithms Unet, PSPnet, Mobilenetv3_deeplabv3+, and Deeplabv3+, MobileViT-Seg has the best overall performance, ensuring a low computational cost while the Mean Intersection over Union (mIoU) reaches 89.79%, the mean pixel accuracy (MPA)reaches 94.46%, the accuracy reaches 94.73% and the detection speed reaches 100.06 F/s. The average accuracy of the proposed picking point localization method reaches 90.80% on 200 raw apple images captured by the camera in real time. The positioning accuracy meets the requirements. In summary, this study provides an efficient technical solution for automated apple picking by implementing an advanced segmentation algorithm combined with precise spatial localization methods, achieving accurate picking point localization in complex orchard conditions. The proposed model can lay the foundation for the picking point localization of apple picking robots.