Abstract: The phenotypic trait of panicle is the key parameter to characterize the growth status, yield, and quality of rice. Accurate phenotyping of panicle is of great significance for field precision management and rice breeding. Unmanned Aerial Vehicle (UAV) image data have been widely used to monitor rice growth status. However, most of studies focused on the vegetative growth stages of rice, and only limited work explored rice panicle phenotyping at the heading and mature stages. Therefore, this study used UAV-based multi-source image data to phenotype rice panicle, analyze the effects of different nitrogen (N) fertilizer levels, growth stages and cultivars on rice panicle phenotyping, and develop the models for monitoring rice panicle coverage, biomass, and lodging. Three field experiments were conducted in Zhuji and Shenyang, China from 2017 to 2018, and a multi-rotor UAV platform equipped with RGB (Red-Green-Blue) and multispectral images was applied to collect rice canopy images. Meanwhile, the ground true values of panicle coverage and lodging were obtained from the RGB images by marking manually object regions, and panicle biomass was measured based on the destructive sampling. The results showed that the panicle phenotyping of rice at different growth stages and N fertilizer treatments was significantly different, and panicle coverage was highly correlated with image features, such as normalized green-red difference index and normalized difference vegetation index. The Support Vector Machine (SVM) combined with the Particle Swarm Optimization (PSO-SVM) accurately identified the panicles from RGB images, and the calculated panicle coverage was highly correlated to the actual marked value with the coefficient of determination (R2) of 0.87. Such a classification model for rice panicles could be applicable to different experimental datasets with the good generalization. Further combination with multispectral reflectance improved the estimation of panicle coverage with the R2 and relatively Root Mean Square Error (rRMSE) of 0.93 and 9.47%, respectively, using the Random Forest (RF) regression model. Fusion of color and texture from RGB images and spectral reflectance from multispectral images improved the estimation of panicle biomass (R2 = 0.84, rRMSE = 8.68%), which outperformed the single RGB or multispectral image data. Further, when the model established from UAV-based multi-source image data in 2017 was used to estimate panicle biomass in 2018, a good estimation result was obtained with the R2 and rRMSE of 0.61 and 15.98%, respectively. Further, the model updating by adding 10% new samples from 2018 to 2017 greatly improved the transferable estimation of panicle biomass, and the R2 and rRMSE were 0.69 and 13.59%, respectively. This indicates that the proposed model for assessing panicle biomass has a high robustness across different planting years. Based on the PSO-SVM classification model, combining color and texture features from RGB images and spectral reflectance from multispectral images accurately identified the panicle lodging with the accuracy of 99.87%. It indicates that UAV-based image features could identify panicle lodging of difference rice cultivars with a close classification accuracy and threshold. The results confirm the feasibility of UAV remote sensing for rice panicle phenotyping, which can provide the decision support for precise crop management and breeding.