Abstract: Different varieties of the same fruit tree can be classified as a fine granularity. Most them, only a single organ of fruit cannot fully extract the overall feature of the plant. It is also difficult to further improve the accuracy of recognition. Multi-organ features can be expected to recognize the fruit tree in current research. However, the inter-organ features can be interfered with each other in the multi-organ feature recognition. It is still challenging to accurately locate and distinguish different organ features. In this study, an organ segmentation (RSA-YOLOv8s) was proposed for the fruit and leaf using YOLOv8s target detection. The three parts of the model were improved, namely Backbone, Neck, and Head. A cross-stage local residual (Residual CSPLayer 2Conv, RC2) module was also designed in the Backbone part. The sensory field of each network layer was used to construct the hierarchical class residual links in each network layer. The dense and small target features were fully extracted after that. The Scale Spatial Pyramid Pooling (SSPP) module was designed in the Neck part. The spatial and scale information of the network were integrated to introduce 3D convolution. The higher-order and multi-scale features were extracted to detect the multi-scale targets. Furthermore, the Asymmetric decoupling detection head (ADDH) module was designed in the Head part. The regression of the model was realized to classify the features using an asymmetric structure. As such, the classification was focused on the center content, while the regression was the edge information. A total of 950 images of 17 fruit tree varieties were selected from the PlantCLEF2022 public dataset. The experimental results show that the precision rate, recall rate, F1 value and average precision of the RSA-YOLOv8s model were 83.2%, 87.9%, 85.5%, and 90.2%, respectively, in the recognition of fruit and leaf organ, which were 5.6, 6.6, 6.1 and 6.7 percentage points higher than those of the original, respectively; By contrast, the precision rate, recall rate, F1 value and mean precision were 88.2%, 91.7%, 89.9% and 94.3%, respectively, in the recognition of single organ, which were 6.4, 6.6, 6.5 and 7.2 percentage points higher than those of the original, respectively; The precision rate, recall rate, F1 value and the mean precision were 78.2%, 84.1%, 81.0% and 86.1%, respectively, in the recognition of leaf single organ, which were 4.8, 6.6, 5.6 and 6.2 percentage points higher than those of the original, respectively. In addition, the improved model was also applied into the 20 date varieties with a total of 3 881 images. A recognition accuracy of 99.1% was achieved, which was 6.6% higher than that of the original model. The finding can provide a strong reference for the organ segmentation and variety recognition using multi-organ features, particularly for common fruit trees.