Abstract: Early growth of rapeseed can be used to assess the performance of seeder, grain yield, crop management, and fertilizer application. Rapid and accurate detection is critical to rapeseed production and yield. However, the manual seedling survey cannot fully meet the operational demands for the extensive or frequent high-precision seedling, due to the low efficiency and subjectivity. In this study, efficient detection and counting models were proposed for the video stream of rapeseed seedlings using unmanned aerial vehicle (UAV) imagery and machine learning. Multi-head self-attention was added to the YOLO series models. The attention was then reduced to irrelevant semantic information. The model was improved to focus on the target object. The basic receptive field block (BasicRFB) module was selected to replace the original spatial pyramid pooling-fast (SPPF) module. One-dimensional convolution was added to the Neck part. The downsampling was then changed to achieve the target of rapeseed seedlings in the image. The efficient fusion of features was also promoted to focus on the key features among other interference factors. The deep simple online and real-time tracking (DeepSORT) was further combined with the cross-line counting to achieve the continuous tracking and target number counting. In addition, the counting model was deployed on edge computing devices. A real-time target counting was designed using a multi-rotor UAV platform. The edge computing device was used to realize the real-time detection and counting of rape seedlings. The targets of rape seedlings were processed in the video stream that was captured by the camera in real time. The experimental results show that: 1) The improved model with multi-head self-attention was significantly focused on the rape seedling area in the image. A better performance was achieved in extracting the target features than before. 2) The detection accuracy of rapeseed seedlings was improved using BasicRFB and the operator of the Neck part. The detection misjudgment of targets was reduced to effectively alleviate the negative impact of invalid targets in the image background. 3) The improved YOLOv5s was achieved in the AP50 and AP95 scores of 93.1% and 67.5%, respectively. Among them, AP50 was significantly higher by 14.82, 26.37, and 3.3 percentage points, respectively, while AP95 was higher by 25.7, 33.9, and 6.7 percentage points, respectively, compared with the classical target detection, such as Faster R-CNN, SSD, and YOLOX. The counting trial of rapeseed seedlings demonstrated that the counting model achieved the maximum precision of 96.34% with an average of 93.75%. Furthermore, the rapeseed counting efficiency exceeded the well-trained operator with an average increase of 9.52 times. In the case of online real-time counting of rapeseed seedlings, the maximum difference in counting precision was 1.87% on the UAV counting platform under different weather conditions. The excellent generalization, counting precision, and efficiency fully met the real-time requirements for rapeseed seedlings. The finding can provide a powerful reference to assess the quality of rapeseed seeding and field management.