Abstract: Object detection can often hold the promising potential to replace the human visual recognition in smart agriculture. However, the small target pests are still challenging to detect for three reasons. Firstly, the pests can move rapidly, which is difficult to for real-time detection. Secondly, the accuracy of the model can depend mainly on the small size, the large number of groups, the uneven distribution, and the occluded pests of each other. Finally, the imbalanced sample can make the recognition more difficult, leading to the low accuracy of detection. It is necessary to efficiently and precisely identify the numerous, unevenly distributed, complex shapes, as well as small and densely packed pests. In this study, a pest recognition (called FieldSentinel-YOLOv8) was proposed using the improved YOLOv8m model. The FieldSentinel-YOLOv8 was improved as follows. Firstly, three detection heads were replaced by two ones, in order to simplify the original YOLOv8 model. The SMO (simplify model operations) was enhanced the fine-grained features for the small targets. The floating-point operations and computational burden were also reduced to streamline the YOLOv8 model; Secondly, the convolutional block attention module (CBAM) was integrated into the YOLOv8. Thus the general feature (such as background) was suppressed to focus more on the pest regions. Thus, the accuracy of the improved model was enhanced to identify the occluded pests. Lastly, the Focal-CIoU Loss was employed to replace the CIoU Loss. The class imbalance was reduced to further improve the accuracy of detection. Many current models depend heavily on the overly idealized datasets, leading to compromise in their accuracy in actual field conditions. Pest monitoring equipment was deployed directly in the field. The datasets were collected under natural environments, indicating the accurate reflection of the real-world presence of pests. The dataset was then taken by the pest monitoring equipment. A series of comparative experiments were conducted to evaluate the performance of the FieldSentinel-YOLOv8 under identical conditions using a self-constructed dataset and various object detections. FieldSentinel-YOLOv8 algorithm also demonstrated the superior performance across most metrics. Compared with the original model, the improved model was reduced the number of parameters by 15.52 M, whereas, there was the an increase in the processing speed to 52.73 frames per second. Moreover, the mAP_0.5 and recall rate of the improved model were enhanced by 2.72 and 7.05 percentage points, respectively. Furthermore, transfer learning was employed to train the FieldSentinel-YOLOv8 model, taking the Pest24 dataset as the source domain and a self-built dataset as the target domain. The trained model was then named the FieldSentinelTransfer-YOLOv8. The improved model was achieved in the better performance of detection after transfer learning. The mAP_0.5 increased by 3.36 percentage points, reaching 77.00%, with the accuracy and recall rates of 69.90% and 77.73%, respectively. Therefore, the FieldSentinel-YOLOv8 can provide the valuable technical references for the accurate and rapid identification of agricultural pests. The high-precision FieldSentinelTransferYOLOv8 model after transfer learning can also offer the technical support for the detection of agricultural pests.