Plant point cloud completion network based on multi-scale geometry-aware point Transformer

Abstract: In the 3D reconstruction of plant seedings, factors such as the occlusion between leaves and the camera's limited field of view often lead to the incomplete of the plant point cloud, which affects the accuracy of plant phenotype analysis. In this study, a plant point cloud completion network based on Multi-scale Geometry-Aware Point Transformer (MGA-PT) was proposed, which adapted to the characteristics of plant point clouds. Firstly, the down-sampling feature extraction module was used to map the raw three-dimensional coordinates to high-dimensional features with the Farthest Point Sampling (FPS) and k-Nearest Neighbour (KNN) algorithm. Not only can this module enhance the feature representation by aggregating local features to reduce the resolution, but also prevent too many input points and reduce the computational burden of the network model. Secondly, a multi-scale geometric-aware module was added in the basic Transformer, which combined semantic features and multi-scale geometric features to form a fusion expression with local information. Multi-head attention mechanism and Feed Forward Network (FFN) were used to extract the semantic features of the plant point cloud, and KNN modules of different scales was used to construct directed local neighbourhood graphs with different resolutions so as to capture various local geometric features and retain geometric relationship information layer by layer. The MGA-PT module promoted the network to have more targeted learning capabilities for different organs such as the leaves and stems of the plants. Then, the dual-path dense point cloud generation module was used to process the input part and the missing part separately. The global features of the plant point cloud were obtained from the encoder, and the missing part of the sparse point cloud and its global features were obtained from the decoder. The sparse three-dimensional coordinates were spliced with the 1 024-dimensional global vector and the two-dimensional grid to facilitate the representation of spatial deformation. The offset of each point was obtained by a folding-based method, so as to expand the point set. It avoided the loss of input point cloud features, and guaranteed a dense point set with realistic distribution. Finally, a dense and complete plant point cloud was obtained by merging the input part and the missing part. The dataset was collected through three-dimensional reconstruction using a low-cost method based on the Structure from Motion (SFM), and the partial point clouds of datasets were obtained by rotation and fixed viewpoint removal. Compared with other models, the method proposed in this study performed well on the plant seedings data. Its average Chamfer Distance (CD) was 0.79×10-4 cm, the Earth Mover's Distance (EMD) was 0.11 cm, and the F1-score was 70.77%. Moreover, it could complete the seedings point cloud of different plant phenotypic traits and missing proportions, which reflected the stability and robustness of the method. It was also effective for missing point cloud completion under actual occlusion. This model can achieve good results in the completion of leaf plants, and this finding can provide a new idea for the completion of plant point clouds.