Abstract: Forestry and fruit industry are two of the most significant components in the agricultural sector, high ecological and economic benefits in China. The forest fruits have been ever-increasingly produced annually. However, it is a high demand for the high-quality forest products, as living standards rise. Accurate assessment of fruit quality can dominate food safety to optimize the transportation and storage for the subsequent grading. Therefore, it is necessary to develop rapid, accurate, and non-destructive quality detection for forest fruits. Particularly, deep learning has been widely applied in the non-destructive quality detection of forest fruits in recent years, due to the automatic features extraction, wide applicability, and end-to-end learning. This review was focused on the research progress of deep learning in forest fruit quality from three aspects: safety, external and internal quality. It was found that Convolutional Neural Networks (CNN) were the most widely used deep learning models in the field of non-destructive quality detection of forest fruits. However, several challenges still remained, including data quality, generalization, and practical implementation. Therefore, some recommendations were then proposed for further application of deep learning. The publicly available datasets should be established to validate the model performance. Future endeavors should explore the application of deep learning-based data augmentation to generate spectral data, beyond only image datasets. Additionally, interdisciplinary collaboration should be fostered to combine microbiology, insect ecology, and plant pathology. Furthermore, standardized evaluation metrics should be developed to validate the scientific soundness of deep learning models, in order to generate various pest and disease data. In terms of models, meta-learning can be expected to design models that can adapt and improve learning strategies. Some challenges are to overcome the influences of origin, batch, environment, and variety on the accuracy of the model. Since most existing studies have been focused mainly on the high model accuracy in recent years, only a little attention can be given to model runtime and detection efficiency. Consequently, a tradeoff between accuracy and efficiency should be explored to develop lightweight models. Moreover, the deep fusion of multiple models is crucial to leverage their respective strengths. In terms of applications, quantitative studies can be concentrated on the severity of fruit pests, diseases, mechanical damage, and appearance defects using semantic segmentation. The time detection of pest infection and mechanical damage occurrence can be improved to increase the collection time and frequency, in order to improve the database for model training. Future research on forest fruit flavors should also be emphasized to facilitate consumer choice. Since deep learning is mostly confined to the laboratory stage, adequate attention can be received in practical production. Therefore, there is a high demand to integrate industry, academia, and research for the intelligent assessment equipment of fruit quality. Some insights can be provided for the further development of deep learning in non-destructive fruit quality assessment, particularly in the digitalization of the fruit industry.