Abstract: Extracting the leaves has been one of the most fundamental researches for the phenotypic traits of plants. However, it is difficult to further improve the accuracy of leaves segmentation, due to the mutual occlusion between leaves in rosette plants. It is still lacking in the leaf edge characteristics, particularly for the small target of young leaves. In this study, an improved instance segmentation benchmark Mask R-CNN model was proposed to realize an accurate segmentation for the plant leaves. The Cascade R-CNN was also introduced to generate the selection of the proposal boxes for the region proposal network. The boxes were first sent into the Head1 convolutional network with a threshold of 0.5, and then the detection was input into the Head 2 convolutional network with a threshold of 0.6, and finally, the detection was fed into the Head 3 convolutional network with a threshold of 0.7. Three cascaded networks were used to gradually increase the threshold, where the output of the previous network was applied to the higher threshold of the next network, indicating the high quality of the detection branch. After the segmentation branch full convolutional layer (FCN), an attention mechanism (SeNet) and a two-layer 3×3 convolution module were added to weigh the features of the segmentation branch, and further to extract the leaf edge segmentation. The segmentation of multi-scale leaves was also adopted for the leaves of different sizes in an image in the test process. The original and enlarged images were input into the model at the same time for the leaves segmentation. The optimal target on multiple scales was achieved using Non-Maximum Suppression (NMS). The ResNeXt101 was selected as the backbone network to extract the characteristics of leaves. An improved model of deep learning was also utilized at the same time. The image enhancement included random mirroring, rotation, enlargement, and size adjustment. There was no longer requiring for the special function library support, indicating the simplified data enhancement. The improved model had effectively reduced the blade occlusion, edge blur, and small target in the blade segmentation. Correspondingly, the Symmetric Best Dice (SBD) was 90.3% in the CVPPP Leaf Segmentation Challenge (LSC), which was 2.3 percentage points higher than that in Mask R-CNN, particularly than 1 percentage points than before. The improved neural network model was suitable for the leaf segmentation. The finding can provide a more accurate leaf segmentation for the plant phenotype using image processing, indicating a high application value in the research field of plant leaves.