Fruit localization for strawberry harvesting robot based on visual servoing

Abstract: As the complexity of the fruits' growth environment, in which the geometric parameters of the fruits and the relative position between fruits and robot need to be structured. The harvesting robots are equipped with custom-made, end-effectors and machine vision system to harvest precisely. Currently, stereo vision and eye-in-hand visual servoing are usually used to locate fruits. The former needs the image matching algorithm which is complex, time-consuming and requires visual sensors with the same parameters and high mounting accuracy. This is one of the difficulties in the field of machine vision. The latter, with simple construction, needs real-time visual feedback which results in large amount of computation and can't get fruits' depth. Under the circumstance of ridge strawberry culture, since the fruits lie on the ridge slope which approximates a plane, the depth of strawberry is almost unchanged during harvesting. Considering these, in this paper, we took the latter method to get the position parameters for harvesting with the eye-in-hand system. Because there was no obstacle between fruits and the end-effector, a Descarte robot which was simple and easy to be controlled but with weak obstacle-avoidance capability was developed. The robot consisted of a gantry walking mechanism, a machine vision based precision positioning mechanism and an end-effector. Firstly, the kinematic equations that describe the position relation between the world coordinate system and the camera coordinate system were deduced. Secondly, the circumscribed rectangle of fruits region's outline was used to describe target fruits region to reduce computation of vision analysis after the image was binarized. Thirdly, to design the visual servoing PID controller, the image Jacobian matrix which relates the end-effector's velocity in the task space to the change rate of feature parameter in the image space was deduced according to the pinhole camera projection principle and the harvesting process. The position of servo motor encoder was recorded with high speed when the motion controller was triggered by the exposure signal of camera to solve the problem that the position at which the camera capturing image inconsistencies with the image caused by the lag of visual feedback. Real-time position information from servo motor encoder was taken to compensate the low frequency of visual feedback. Fourthly, the equation of harvesting position parameters was deduced, from which the distance was obtained between the top of the target fruits' region and the end-effector and the width of the target fruits' region. With these parameters, the end-effector's lifting height and opening width was known. Fifthly, according to different centroids of target fruits' region in the image while the camera moved to different positions during the visual servoing process, the depth of target fruits' region was obtained on the basis of structure from motion. The test of fruit localization was respectively conducted when the angle between the ridge plan and the image plan was ±10, ±5, 0 degrees while the target fruits region contained 1, 2, 3 strawberries. Because the relation between harvesting position parameters and the depth of target fruits' region can be drawn that the accuracy of harvesting position parameters met the harvesting requirements when the error of the fruits depth was within a certain range, and only the depth error was obtained to evaluate the accuracy of harvesting position parameters in the test. The results showed that the angle, the number of strawberries of the target fruit region and their interaction had significant effects (P=0) on the localization time which was between 0.633 s and 0.886 s. The angle and the two elements' interaction had significant (P=0) effects on the relative error of the fruit depth which was in the range of -4.34% ~ 0.95% when the angle was invariant. The localization time increased with the number of strawberries of the target fruit region as the increase of the target fruit region centroid which increased with the number of fruits decreased the controlling speed. That the standard error of the relative error of the fruit depth increased with the number of strawberries of the target fruit region showed that a wider target fruit region led to a lower robustness of the relative error of the fruits depth.