Abstract: A rapid and accurate evaluation of crop yield at large scale is very critical to the planning and management of crop market. The prediction of crop yield includes the extensive manual sampling or the use of remote sensing images at present. Particularly, the remote sensing is preferred to predict the large-scale crop yield, providing a large number of satellite images with long time and high spatial resolution. Much effort has been made on the crop yield prediction using remote sensing images and deep learning. However, the yield generally presents nonlinear spatio-temporal characteristics, due mainly to the growth status of crops changes with the time. A single deep learning cannot make full use of crop growth characteristics in multi-temporal images. In this study, a Convolutional Neural Networks - Gated Recurrent Unit (CNN-GRU) estimation model was proposed to extract rich spatial-spectral features from multi-spectral remote sensing images. The time dependence in the phase remote sensing image was also adaptively learned among the various stages in the growth period of winter wheat. As such, the growth characteristics of winter wheat were integrated from multiple scales, further to predict the yield. Furthermore, the main production areas of winter wheat in the country were taken as the research area, including 1 713 districts and counties. The forecast dataset of winter wheat production was finally constructed using the statistical output of winter wheat, MODIS surface reflectance, day and night surface temperature images, and land cover data from 2001 to 2018, thereby to verify the performance of CNN-GRU production estimation model. The results showed that: 1) The annual average root mean square error (RMSE) and mean absolute error (MAE) of the CNN-GRU production estimation model from 2016 to 2018 were 818.3 and 560 kg/hm2, respectively, where the annual average RMSE was reduced by 20.13%, 18.81%, 29.51%, 34.84%, and 36.57%, respectively, compared with the CNN, GRU, Support Vector Regression (SVR), Random Forest (RF), and Decision Tree (DT). Therefore, the CNN-GRU yield estimation model can be expected to accurately predict the winter wheat yield, further to effectively extract the space-spectrum-time information in the entire growth period of winter wheat from multi-temporal remote sensing images. 2) Six time windows were divided during the whole growth period of winter wheat, including the sowing-emergence, tillering-overwintering, green-rising, jointing-booting, heading-flowering, and filling-maturity period. Among them, the highest accuracy of CNN-GRU estimation model was found in the filling-maturation period. Specifically, the annual averages of RMSE and MAE were 817 and 556 kg/hm2, respectively, and R2 was greater than 0.7. However, the annual averages of RMSE and MAE during the heading-flowering period were 823 and 560 kg/hm2, respectively, where the annual average RMSE was 0.7% lower than the filling-maturation period. Therefore, the proposed CNN-GRU yield estimation model can accurately predict the output of winter wheat in the middle and late stages of growth period, particularly for the national yield of winter wheat two months in advance.