Hand-eye calibration method based on Sylvester equation deformation for lychee harvesting robot

Abstract: Establishing the correlation between vision system and robot is the key and premise for the fruit harvesting robot based on binocular vision to successfully harvest fruit. And the robot with eye-in-hand vision system can use the active vision to search harvesting targets in a great scope and find out the position of picking points related to the robot’s coordinate in real time. For the accurate hand-eye calibration of the lychee harvesting robot with eye-in-hand vision system, a hand-eye calibration method based on optimization solution of the homogeneous matrix equation was proposed. Binocular camera was installed on the end of the robot's manipulator. The calibration board was placed in front of robot and the calibration board was stationary relative to the robot. By controlling the robot’s movement, binocular camera can observe the calibration board from multiple positions. Then the relative position relationship data between binocular camera and calibration board can be collected by stereovision location. Also, position data of manipulator’s end relative to robot coordinate system can be calculated using the robot kinematics. These data above can be used to establish an approximate equation of hand-eye calibration. Sylvester equation deformation was used to transform the approximate equation into a linear equation that made the problem into a linear optimization problem with some constraints. Lagrangian-Multiplier method was applied in changing the constrained optimization problem to a global optimization problem, and the conjugate gradient methods was a good choice for solving this global optimization problem. The result calculated by traditional method was used as the initial value that a more accurate solution can be solved out by iteration. This hand-eye calibration method was a module of our software “Stereo Vision Lychee Picking Point Recognition and Localization System” which had already been implemented using C++ and OpenCV. The software ran in an industrial control computer cooperated with the harvest robot and binocular vision system and were tests for three experiments. The first experiment was the robot’s mechanical arm movement influence on the ranging error of binocular vision system. This experiment showed that when the binocular cameras had been installed on the terminal of robot’s mechanical arm, the mean ranging error of binocular vision was associated with the movement times of robot’s mechanical arm. When the binocular cameras’ working distance was about 1000 mm, the initial mean ranging error of binocular vision would be 0.55 mm. And then, with the increase of movement times of robot’s mechanical arm, the mean ranging error of binocular vision presented a rising tendency. When the movement times of robot’s mechanical arm was more than one hundred, the mean ranging error of binocular vision would be more than 2 mm. Hence, the best movement times of robot’s mechanical arm used to hand-eye calibration can be 15. This experiment also showed the probable cause of repeat joint positioning error of robot with eye-in-hand visual system. The second experiment was the repeat joint positioning error of the robot with eye-in-hand visual system. The mean repeat joint positioning error of the robot generated by our hand-eye calibration method base on Sylvester deformation equation was 2.929 mm, which was better than the 4.442 mm error generated by traditional method . Meanwhile, the standard deviation of repeat joint positioning error of the robot was 0.454 mm, which was better than the 0.554 mm standard deviation of error generated by traditional method. The second experiment showed our hand-eye calibration method was better than the traditional method in the aspect of precision. At last, we used this harvesting robot to harvest lychee in a real lychee orchard. This experiment can show the reliability of our hand-eye calibration method in the natural environment. We tested 51 times, the ratio of successful picking was more than 76%. Moreover, when the software accurately recognized and located the lychee picking point, the ratio of successful picking could reach 92.3%. This experiment showed that the proposed hand-eye calibration method and the transformation result calculated using our method had a strong reliability and could satisfy the actual picking requirements of lychee harvesting robot.