Abstract: Camellia oleifera fruits are required for the high harvesting efficiency during the non-flowering and fruiting period. In this study, the key points of lateral branches was located to reduce the tree damage for the efficient harvesting. Firstly, the dataset was constructed to label the lateral branches in the segments. The labelling was then divided into five categories, including cof (camellia oleifera fruits), br_con (coniferous branch), danger (vibration danger area, the area of forked stems), br_pro (vibration-prioritised picking area), and br_m (the area of transitional branches, in which harvesting was not considered). Segmented labelling of side branches was reconstructed from the side branches, particularly for the subsequent priority detection of key points. UNet network was selected as the segmentation network. The CloFormer attention mechanism was then added into the UNet, named Clo-UNet. As such, the high-precision segmentation was realized on the lateral branches. The field experiment was carried out to verify the detection. The mIoU values of the continuous fruiting branch br_con, the vibration hazardous region danger, and the vibration priority region br_pro reached 85.36%, 86.37% and 81.29%, respectively, while the mPA values were 94.97%, 96.17% and 89.48%, respectively. The bifurcation points of lateral branch were avoided to reduce the end of the branch during the actual harvesting. The actuator and robotic arm were reduced the possibility of damage at the branch bifurcation point. The Clo-UNet was further designed for the vibration picking priority of key points, named as the Clo-UNet-Point. The center of mass was selected in the br_pro region as the vibration picking point, in order to ensure the maximum transmission of the actuator's excitation force during harvesting. If the br_pro was absent, the detection was then determined to remove the br_con region. The reason was that the br_con was the continuous fruiting branch, thus bending into a bow shape under the gravity of the camellia oleifera fruits. The convexity was firstly used to determine the shape of the region, if the convexity was greater than 500. A bow shape was set as the rectangle for the bow shaped region, whereas, the straight line was fitted to the region. After that, the pixel points of the region to the straight line were calculated as the harvesting point. The minimum distance from the pixel points of the region to the straight line were taken as the harvesting point. The harvesting point of the rectangular region was calculated in the same way as br_pro. Finally, all the images were observed on the validation set. The key point was detected in the case of segmented branches. The key point was 100% on the branches. The pickup point recognition was summarized to determine the harvesting priority of YOLOv8n-Point, compared with the Clo-UNet-Point. The whole detection of YOLOv8n-Point was taken about 1.94 s in the traditional image processing, such as the skeleton extraction. By contrast, the Clo-UNet-Point was 0.15 s. The harvesting point was beyond the branch, while the key point was detected at the end of the branch unsuitable for the transmission of force during vibration. The finding can lay the theoretical foundation on the automatic vibration harvesting for the non-flowering and fruiting contemporaneous classes of camellia oleifera fruits.