Stem-leaf segmentation and phenotypic trait extraction of tomatoes using three-dimensional point cloud

Crop phenotyping is the quantitative assessment of complex plant traits. The crop phenotypic parameters can greatly contribute to crop breeding, product development, and quality evaluation in modern agriculture. However, the traditional and manual measurement can be time-consuming and labor-intensive with low data accuracy. In this study, an intelligent measurement system was developed to automatically detect the phenotypic parameters of tomatoes in a greenhouse with high throughput using a mobile robot platform. A Kinect V2 depth camera was first installed at the end of the robot arm, and then a multi-perspective point cloud was collected via the movement of the robot arm. An automatic and accurate inspection was conducted in the greenhouse for the automatic collection of the tomato point cloud. The tomato point cloud was filtered and registered to obtain the complete plant point cloud. In the registration process, the end joint pose recorded by the robot arm was used to calculate the rotation and translation matrix of point cloud registration using the change table relationship of the pose construction. After that, the tomato point cloud skeleton was extracted to realize the stem and petiole segmentation using Laplace contraction. An experiment was also carried out in the greenhouse of the National Experiment Station for Precision Agriculture in Beijing in March 2022. Multi-view point cloud collection and phenotypic parameter extraction were performed on 10 tomato seedlings at 7, 15, and 25 days after transplanting to the greenhouse. The accuracy, recall rate, F1 fraction, and average overall accuracy of stem and leaf segmentation were 0.84, 0.91, 0.87, and 0.92, respectively. The MeanShift clustering and regional growth were combined to divide the leaf and leaf petiole for the better segmentation of tomato leaves. Among them, the regional growth algorithm presented an excellent performance to segment the leaf and leaf petiole, whereas, the MeanShift algorithm was to segment the different leaves. The accuracy rate, recall rate, F1 score, and average overall accuracy of leaf segmentation were 0.92, 091, 0.91, and 0.93, respectively, indicating better performance than the regional growth and MeanShift algorithms alone. The parameters of plant height and stem diameter were measured by the plant point cloud after registration, and the parameters of leaf inclination were measured by the skeleton. Finally, a greedy projection triangulation algorithm was selected to convert the leaf point cloud into the triangular mesh for the parameters of leaf areas. The determination coefficients of plant height, stem diameter, leaf inclination, and leaf area were 0.97, 0.53, 0.90, and 0.87, respectively, and the root mean square errors were 1.40 cm, 1.52 mm, 5.14°, and 37.56 cm2, respectively, compared with the measured values. Specifically, the larger error of stem thickness parameter can be attributed to the distance measurement error of the depth sensor for the very thin stem of the tomato plant at the seedling stage during automatic measurement. Subsequent research can be required to accurately locate the tomato stem. A vernier caliper was installed at the end of the robotic arm to perform the contact measurement, in order to effectively improve the accuracy of the stem diameter parameter. The finding can provide technical support to the high-throughput, accurate and automated measurement for the phenotypic parameters of greenhouse tomatoes.