Abstract: Sorting winter jujube is one of the most essential tasks after harvesting. An effective sorting and grading can be used to enhance the brand recognition of winter jujubes. The current sorting task of winter jujube work still relies mainly on manual labor. Alternatively, large-scale sorting equipment cannot be afforded for the individual jujube farmers, due to its high price. Fortunately, deep learning has been widely applied in the field of agricultural product detection. It is also expected for the intelligent and automatic sorting of winter jujubes. However, there is still a research gap in the market for the small-scale devices and algorithms suitable for winter jujubes sorting. Furthermore, the existing equipment in winter jujube sorting is also confined to the low accuracy, slow speed, and high cost. In this study, an improved Faster R-CNN model was proposed to identify the external defects of winter jujubes for the small-scale sorting. Firstly, the original VGG16 feature extraction network was replaced with the ResNet50. An SE module was then added into ResNet50 to explicitly model the interdependencies among channels, where the network was adaptively recalibrated the feature responses of each channel. As such, the network was used to focus more on the useful features while suppressing irrelevant ones, thereby enhancing the representation capability of features. Secondly, the feature pyramid network (FPN) was integrated to fully extract feature information at different scales. Finally, the original non-maximum suppression (NMS) algorithm was replaced with the improved Soft-NMS algorithm to reduce the issue on the mistakenly deleting targets of defect detection in the detected images, further improving the accuracy of winter jujube defect identification. The experimental results show that the improved Faster R-CNN model shared a mean average precision (mAP) of 91.60% for the winter jujube defect detection, with a detection speed of 17.5 frames per second. The mAP was 14 percentage points higher than SSD, 11.32 percentage points higher than YOLO v3, and 5.94 percentage points higher than YOLO v5. When the improved Faster R-CNN network model was deployed on the detection platform to sort the winter jujubes of different qualities, the recognition accuracies for Good fruit, Mechanical injury fruit, Cracking or splitting fruit, Insect fruit, and Disease fruit were 94.71%, 96.99%, 99.06%, 93.82%, and 99.05%, respectively. Although the detection time of two-stage object detection networks was longer than the rest, the impact of detection speed was neglected with the support of high-performance computers. The improved Faster R-CNN network model can effectively and rapidly identify the appearance of winter jujubes, thus reducing the false detection rate in the winter jujube detection. The research finding can provide the promising ideas and theoretical support to the mechanization of the winter jujube industry.