Fruit bunch measurement method for cherry tomato based on visual servo

As the fresh cherry tomato, largely produced and consumed in China, needed an increasing manual-picking cost these years, the automatic harvesting machine was expected to replace the manual labor to implement the intensive work. Accurately identifying and locating the mature fruit bunches, was a key technique of harvesting robot research. The existing research could be classified into active and passive detecting method, among which the active method was becoming more mainstream. In this paper, a new vision system for cherry tomato's automatic harvesting was designed based on laser ranging and vision servo, and the system included a camera, a laser sensor and a manipulator as the servo unit, among which the camera was fixed coaxially ahead of the laser sensor, and could slid up and down driven by a cylinder. When the camera slid down, the laser sensor was triggered to measure the distance between fruit and camera's view system. Through analyzing the color feature of the image acquired from the camera, the R-G color model was adopted to intensify the difference between target fruit and background. According to the column pixel grey statistics, the candidate area of fruit bunch was selected from the R-G image, with a view of decreasing the whole image processing and improving recognition accuracy. Then the CogPMAlignTool contained in the Cognex VisionPro image processing classlib was used for the fruits' identifying from the bunch, with the single fruit's template scaling range (0.8, 1.2), rotating angle range (-π, π) and acceptable threshold 0.36. According to the image coordinate of the periphery fruits and the coordinate transformation between the camera and the manipulator, the stereo coordinate was estimated based on the camera imaging model, which was considered as the initial position for targeting the fruit based on the vision servo. The transition matrix between the camera and the manipulator was determined through the hand-eye calibration. According to the deviation between the fruit's center and the image center, the base joint and the forearm joint were controlled to change the posture of the camera based on vision servo algorithm, so that the 2 centers in the fruit and the image could coincide approximatively. After aiming the fruit, the laser sensor was triggered to measure the distance between the vision system and the fruit, and the accurate coordinate of the periphery fruits could be obtained on the basis of the distance and the gesture of manipulator. Furthermore, the width and the length of the bunch were calculated after the coordinates of 4 periphery fruits were measured, which would be the necessary parameters to guide the robot's grasper to case the bunch from the bottom up with the fruit bag and cut the stem. The test result showed that the average successful identification rate of single fruit from the bunch was 83.5%, and the rate would be better, if the fruit bunch had more regular shape, or the bunch stem was closer to the view center of vision unit; and through the vision servo control to aim the fruit center, the average deviation between the fruit center and the image center was 8.38 pixels. Finally, the measuring error of bunch length was 8.25 mm, and that of bunch width was 5.25 mm.