Abstract: This study aims to realize the intelligent thinning operation on the apple tree. An inflorescence thinning was also proposed for apple trees at the single-branch level in modern orchards. Firstly, the YOLOv8-Seg model was improved as follows. The GCT (Gaussian Context Transformer, GCT) module was added to Backbone. The segmentation performance of the occluded targets was improved to introduce the global context information for the importance of channels. Efficient Multi-Scale Attention (EMA) mechanism was added inside the C2f module. The Neck portion of the three detection heads was better focused on the multi-scale features using parallel subnetwork structure and cross-space information aggregation. Secondly, the improved YOLOv8-Seg model was used to segment instances for four types of targets: buds, inflorescences, open flowers, and flowering branches at the single-branch level. Finally, the polynomial fitting curves were used to represent the flower branches on images after instance segmentation. At the same time, the distance between inflorescences was calculated for the inflorescence thinning. 600 images of three apple varieties were collected, including Fuji, Gala, and Ruixue. An image dataset of the flower branch was then created using these images. The segmentation model and thinning decision were tested using the above dataset. The training was then conducted on the improved YOLOv8-Seg model. All losses were tended to stabilize. Then the model was converged after 1000 epochs. The mAP gradually increased to eventually stabilize during training. There was a small fluctuation of mAP during overall training. The model was then converged to the local optimal solution during training, indicating a relatively stable performance. The ablation experiment was used to verify the two improvements of the model, namely the GCT module and the EMA mechanism. The results showed that both improvements significantly enhanced the original model. 50 images of Fuji, Gala, and Ruixue apple varieties were randomly selected to verify the robustness of the improved YOLOv8-Seg model. 150 images in total were used to test the segmentation performance of different apple varieties. The improved YOLOv8-Seg was used to test the segmentation. The results showed that there were small differences among the three varieties in the multiple evaluation indicators, indicating the high robustness in segmenting apple flower branch images. Comparative experiments were conducted to evaluate the performance of the improved YOLOv8-Seg model, particularly with the mainstream segmentation models, such as Mask R-CNN, YOLACT, SOLOv2, and YOLOv9-C. The better segmentation performance of the improved model was achieved, where the mAP values were 10.8, 12.3, and 9.1 percentage points higher at the mask level, respectively, compared with the Mask R-CNN, YOLACT, and SOLOv2 models. The segmentation performance of the improved model was slightly inferior to the YOLOv9-C, but the improved model was significantly lower in complexity, weight size, and parameter quantity. The precision, recall, and mAP of the improved YOLOv8s-Seg model on the self-built dataset at the pixel level reached 89.9%, 89.5%, and 91%, respectively, which were 6.5, 4.1, and 5.8 percentage points higher than the original model, respectively. There was little difference between the inflorescence thinning and manual decision. The improved model can be applied to the task of inflorescence thinning at the level of a single branch. The finding can also provide technical support to the apple flower thinning.