Apple detection and counting using real-time video based on deep learning and object tracking

Abstract: Yield estimation for apples is a key for predicting stock volumes, allocating needed labor, and planning harvesting operations. Manual visual yield estimation for a small number of trees to predict the number of fruits in an orchard has traditionally been employed, resulting in inaccurate and misleading information. Therefore, an automated solution for orchard yield measurement is urgently needed. Detection and counting of fruits infield based on machine vision coupled with advanced machine learning algorithms is a key to realizing orchard yield measurement automatically, which can provide baseline information for better production management. Therefore, this study aims to develop an automated video processing method to realize the automated detection and counting of apple fruits in an orchard environment with a modern vertical fruiting-wall architecture. This study proposed a fruit counting method based on a lightweight YOLOv4-tiny network and Kalman filter algorithm toward this end. ‘Fuji’ variety was selected, which is widely planted in modern planting patterns. 800 images and 10 videos of apple trees were acquired using a remote-controlled car equipped with a Realsense D435 camera. Firstly, fruits in the orchard video were detected using the trained YOLOv4-tiny model. Secondly, all detected apples would be predicted based on the Kalman filter algorithm. Subsequently, all predicted and detected apples in the subsequent frame would be optimally matched based on the Hungarian algorithm of the Euclidean distance and Intersection over Union (IoU). Successfully matched fruits would be added to the tracked track, based on which the corresponding Kalman filter was updated. The trajectory that failed to match would be temporarily saved until the match failed for 30 consecutive frames, while the detection target that failed to match was regarded as a new fruit. Finally, the fruit digital ID would be assigned based on the appearance sequence of the fruit in the video frame to realize the fruit count. In order to prove the superior performance of the deep learning network trained in this study, YOLOv3-tiny was chosen to use the same test dataset for comparison with YOLOv4-tiny. The test results showed that the Average Detection Precision (ADP) based on YOLOv4-tiny reached 94.47%, which was 1.76 percentage points higher than that of YOLOv3-tiny. Besides, YOLOv4-tiny only took 0.018 s on average to detect fruits in one image with the resolution of 720×1 080 pixels, which was 0.009 s faster than YOLOv3-tiny. It could be seen that YOLOv4-tiny could achieve high-precision detection of fruits at a faster speed, which provided a good foundation for fruit counting. The Multiple Object Tracking Accuracy (MOTA) and the Multiple Object Tracking Precision (MOTP) based on Kalman and improvement Hungarian algorithms were 69.14% and 75.60%, which were 26.86 percentage points and 20.78 percentage points higher than the indicators based on Kalman and the unimproved Hungarian algorithm, respectively, indicating the reliability of the tracking algorithm. Furthermore, an average precision of 81.94% and a determination coefficient of 0.986 with counting performed by manual observation were reached in 10 orchard videos. The method developed in this study can effectively feedback the detection and counting results of apples in the orchard for growers, provide technical reference for the production measurement research of modern apple orchards, and provide scientific decision-making basis for intelligent management of orchards.