Current status and future perspective of the application of deep learning in plant phenotype research

Abstract: Accurate plant phenotyping is important for gaining a fundamental understanding of phenotype-genotype-environment interaction and is also critical for plant breeding and agricultural precision management. With the development of accurate and high-throughput plant phenotyping techniques, big phenotypic data of various plants especially image data can be collected. There is an urgent need to develop effective approaches to dealing with large-scale image data analysis to explore the biological and physiological mechanisms which can be eventually used from the laboratory to the field. This research was entering a new era called 'smart phenomics'. Deep Learning (DL) provided an opportunity to extract useful traits from the complicated phenotypic dataset, which could bridge the knowledge gap between genotype and phenotype for fundamental research and engineering applications in a breeding program and precision farming. Recently, a series of phenotyping related research supported by DL had been published all around the plant fundamental mechanism as well as the agricultural engineering applications. This study investigated the latest publications focused on phenotyping relating to the following algorithms: Convolutional Neural Network (CNN), Restricted Boltzmann Machines (RBM), Auto Encoder (AE), Sparse Coding (SC) and Recurrent Neural Network (RNN), both of the achievements and problems were introduced and summarized in the following aspects. The published researches involved the phenotypic identification and classification over various crops from tissues, organs, and plant scales singly or combined. Not like DBN, SC, or other earlier algorithms, CNN could extract the features without image preprocessing or feature design, its capability also grew rapidly since it was proposed and now had been the first-choice for image identification and classification scenarios. While deep learning applications in biotic/abiotic plant stress analysis mainly focused on the identification and classification of different phenotypic traits of various common crops under typical stresses. Recent studies CNN showed the most potential capability for stress identification and classification, and the predictions by CNN was an irrelevance to the type of stress. Studies also found that the qualitative analysis of abiotic stress could be diagnosed by transfer learning to reduce training time without affecting the prediction capability of the model, especially network architectures with mature applications scenarios manifested stable performance in terms of adaptability and migration based on CNN or integrated with CNN, Besides, yield prediction accuracy had been greatly improved through color, geometric shapes, textures and multiple phenotypic coupling features, which could be divided into the following three scenarios, including fruit yield prediction based on fruit identification and counting, field crop yield prediction by Unmanned Aerial Vehicle (UAV) remote sensing and multi-dimensional yield prediction based on multi-scale, multi-source, and multi-factor data. Moreover, deep learning had shown the potential for precision breeding and precision management. The precise identification of crop-specific phenotypes is the basis for accurate breeding and phenotypic management, and it can be summarized as two categories, including quantitative index counts after target detection and target segmentation under complex field conditions. In summary, a large number of proposed network architectures applied in plant phenotyping have been reviewed, and future efforts should be made on improving the efficiency and accuracy in production scenarios. Finally, the trend and future perspective in the multi-disciplinary research field of deep learning in plant phenotype research were also discussed.