Abstract: Pine wilt disease (PWD) is known as the cancer of pine trees, however, the detection of discolored dead trees often results in false detection and missed detection. In this study, an adaptive multi-scale feature fusion network was proposed to optimize the recognition accuracy for the PWD. The backbone part of the lightweight network (EfficientNetv2) was used as the feature extraction. The spatial pyramid pooling fusion (SPPF) module in YOLOv5s was retained to compress the number of model parameters, to reduce the amount of computation; Secondly, the convolution block attention module (CBAM) was added to YOLOv5s to strengthen the attention to target disease, and the PANet was replaced with BiFPN to optimize the accuracy of discolored dead trees. The feature weight information was introduced to enhance the fusion of features at different scales. Finally, the CIoU loss function was replaced by the SIoU loss function to improve the model accuracy. Direction matching between the real and predicted frames was considered to enhance the model’s convergence. The results showed that the accuracy, recall and mAP_0.5 of the improved algorithm in the modeling area reached 92.95%, 94.88% and 94.21%, respectively, which was 4.31%, 5.60%, and 5.13% higher than that of the original model. The number of parameters and computation were reduced by 22.93% and 64.19%, which were 77.07% and 35.81% of the original YOLOv5s, respectively. At the same time, the network processing speed of the improved YOLOv5s model was increased by 45.66% to 136.05 frames/s, and the model was more real-time. The ablation test proved that each improvement measure based on YOLOv5s improved the performance of the original model, and all four measures were necessary. Compared with SSD, Faster R-CNN and YOLO series object detection models, the improved model showed significant advantages in terms of recall rate, detection speed, number of parameters and model weight. The improved model showed 95.25% and 93.17% of the single-class average detection accuracy (AP) for abnormally discolored pine trees and dead pine trees, respectively. In addition, the frame rate and weight file size of the improved model were 136.05 frames/s and 11.90 MB, respectively, and the F1 score was 93.91%. The average detection accuracy of the improved algorithm for two types of targets in the natural forest area was 1.07% higher than that in the modeling area. The recall rate of dead pine trees in plot 8 reached 95.88%, which was 1.00% higher than that in the modeled area. After the detection of the two types of targets in the test area, the average F1 score of the model was 93.68%, which was roughly equivalent to the modeling area. This detection method was suitable for the rapid and accurate detection of abnormal discolored pine trees in natural scenes, which was of great practical significance for improving the intelligent level of pine wood nematode disease prevention and control.