Recognition of the apple in panoramic images based on improved YOLOv4 model

Yield forecasting is one of great significance for decision-making in the apple industry, including labor hiring, harvesting, and storage allocation. Traditional forecasting of apple yield relies mainly on the manual counting of some of the apple trees to estimate the yield of the entire apple orchard. The inaccurate prediction cannot fully meet the large-scale production in recent years. Therefore, it is a high demand for a more accurate and labor-saving way to forecast apple orchard yield. Artificial intelligence in smart orchards can be expected to combine with traditional orchards in the development of the apple industry. The accurate recognition of apples is one of the key technologies to achieve the intelligent yield estimation of apple orchards. However, the shading between apple trees has posed a great challenge to apple fruit identification at present, due to the dense cultivation mode in apple orchards. The repeated capture of apple fruit images can lead to inaccurate fruit counting in the image-collecting mode in each fruit tree. In this study, a panoramic image of apple recognition was proposed using an improved YOLOv4 and threshold-based bounding box matching and merging algorithm. Panoramic photography was used to collect the images of apple fruit trees. Firstly, the Spatial-Channel Sequeeze & Excitation (scSE) attention modules were added to the Resblock module of the backbone of YOLOv4. Some convolutions in the PANet module and YOLO Head module were replaced by the depthwise separable convolutions. The number of output feature channels of depthwise separable convolutions increased to enhance the feature extraction capability, but to reduce the number of model parameters and computation. The panoramic image was segmented into several sub-images. The improved YOLOv4 model was selected to recognize the apples in the sub-images. A comparison was performed on the recognized data of different network models, such as the Faster R-CNN, CenterNet, YOLOv4, YOLOv4-Lite, YOLOv5-l, and YOLOv5-x for the panoramic images of apple trees. The improved YOLOv4 network model achieved a precision rate of 96.19%, a recall rate of 95.47%, and an AP value of 97.27, which were 1.07, 2.59, and 2.02 percentage points higher than the original YOLOv4 model. Secondly, the bounding boxes of the apples in the recognized sub-images were matched and merged by the threshold-based bounding box matching and merging, in order to realize the recognition of panoramic images. The validation experiments determined that the thresholds of 3, 1, and 45 were used for the D1, D2, and D3, respectively. A better performance was achieved in the precision rate of 96.17%, a recall rate of 95.63%, an F1 score of 0.96, and an AP value of 95.06%, which were higher than each evaluation index for the direct recognition of panoramic images of apples. As such, the panoramic image recognition was obtained to merge the sub-image recognition using the improved YOLOv4 model. The higher evaluation index and better recognition were achieved in the apple recognition of panoramic images under natural conditions. The finding can provide a new strategy to recognize apple fruits for the intelligent measurement of orchard yield.