Abstract: Intelligent insect detection and alarm lights can timely monitor the pests in the intelligent production of smart agriculture at present. However, some pest alarm lights cannot provide reliable data support for pest detection and reporting tasks, due mainly to the low accuracy of detection and identification. This study aims to improve the classification and recognition to fully meet the requirement of pest situation lamps using deep learning. The mask RCNN instance segmentation model was selected as the basic detection framework. A network model was constructed to recognize the multiple similar graphs using ResNet50-FPN convolutional neural network (CNN). According to Faster RCNN, a mask branch was added for the instance segmentation. An anchor frame was used for the classification and regression. The pixel segmentation and classification were added for a more accurate classification. The framework of object detection was obtained with the best detection accuracy in fine-grained classification. Nevertheless, the main difficulty of the model was the nonuniform size of the classification target. Much more interferences of similar insects were easy to produce the target adhesion, due to the high density. Mask RCNN model relied normally on the dense anchoring scheme, the predefined scale and the aspect ratio to sample uniformly. The improved predefined anchor frame cannot solve the adhesion of targets at various scales. Moreover, the difference between non-targets and targets cannot be distinguished, although there was a distinction among different subclasses of the same category. Five datasets of target pests were established in this case. A small proportion of the miscellaneous insect dataset was added to reduce the fraction of miscellaneous insects. In adhesion, soft NMS was used to replace NMS, and DeAnchor was adopted to learn each target area. The detection frame was then fitted to the target during model prediction and recognition. The detection frame was further reduced to exceed the area or expose the surroundings during prediction. The adhesion and recognition rate were greatly improved after three steps. In non-target misidentification, the NDCC was added to exclude the unknown miscellaneous insects using the novelty score. The reason was that there were many more types of non-targets similar to targets, leading to the high misidentification rate of non-targets. The optimal model was achieved with the highest recognition accuracy of 96.1% for the multi-insect pictures with 10 insects per picture, and 93.5% for multi-insect pictures with 20 insects per picture. The accuracy rate of multi-insect image recognition reached more than 90.6% for the identification number of 50 insects per image. The adhesion was basically solved in the larger adhesion target in the picture. Non-target recognition and misrecognition were improved in the pictures containing no-target insects. Moreover, the error of detection rate reached the lowest 9% in the recognition of 20 pure non-target images. The error of detection rate was also reduced to less than 15% in the recognition of 40 mixed target images. The functions of intensive and novelty detection were added to the existing classification model, in order to enhance the detection ability of dense areas. The non-target misidentification was improved for the higher classification and recognition accuracy of pest classification in the actual environment.