Recognizing pig behavior on posture and temporal features using computer vision

Monitoring pig behavior has been one of the most important steps in the process of pig breeding management. Computer vision technology has been used to quickly and conveniently detect the activities of pigs for the welfare in recent years. In this study, a monitoring system of pig behavior was proposed for automatic recognition using improved computer vision. The five steps wereas follows. 1) The monitoring video was collected in a pig house using a network camera; 2) The single bounding box regression and the pose of pigs were extracted by the YOLOv5s and lightweight OpenPose algorithms, respectively; 3) A Simple Online and Realtime Tracking (SORT) approach was used to match the same pig target in the multiple video frames; 4) The spatiotemporal map of pig skeleton was generated to extract 200 frames of continuous pose images; and 5) A spatial temporal and graph convolutional networks (ST-GCN) model was selected to process the spatiotemporal map of pig skeleton for the prediction of behavior categories. Therefore, the video data was first collected on the herd of pigs. A LabelImg software was then used to mark the object detection data set, whereas, the Labelme tool was to mark the keypoint data set. A total number of more than 700 pig behavior video clips were extracted for the data sets. The pose extraction script was used to automatically extract the pig pose information from the behavior video clips, where the behavior data set was generated in kinetics-skeleton format. The object detection data set was selected to train the pig object detection model using the YOLOv5s algorithm. The keypoint data set was used to train the pig pose estimation model using the Lightweight OpenPose algorithm. The pig behavior data set in kinetics-skeleton format was used to train the pig behavior recognition model using the ST-GCN algorithm. All models were trained using the PyTorch deep learning framework on an NVIDIA Tasla T4 GPU. Some indicators were selected to evaluate the models after training. The precision of the target detection model was more than 97%, while the average accuracy and recall rate of the pig pose estimation model was more than 93%. The recognition accuracy for the standing, walking, and lying behavior were 93.3%, 73.3%, and 93.3% in the pig behavior recognition model, respectively, where the average accuracy was 86.67%. The results show that the ST-GCN approach was feasible for pig behavior recognition. A graphical interface system was then developed for the pig behavior recognition by the QT framework. Furthermore, the LibTorch library was used to inference the models in the behavior recognition system. The system was running on the NVIDIA GTX 1080Ti GPU, with 14 ms taken by the YOLOv5s model inference, 0.05 ms by the SORT tracking, and 65 ms by the pose estimation model inference. The average processing frame rate was about 12 FPS. Every 200 frames of continuous pig posture were extracted by the program, and the behavior recognition reasoning was carried out once. The behavior recognition inference of a single pig was taken at about 8ms, and the average time of updating behavior recognition was about 17 s. The finding can provide a new idea for the behavior recognition of pigs.