Abstract: The picking maturity of olive fruit varies greatly in different uses. However, the commonly-used manual picking cannot fully meet the picked fruit in the best picking period, due to the low picking efficiency and high cost during olive fruit picking. The purpose of this study was to determine the distribution areas of fruits in the different maturity under the natural environment, in order to realize the automatic picking of olive fruits. Firstly, the phenotypic characteristics of olive fruit at maturity were determined to develop the new standard of ripeness, according to the recommendations of the International Olive Council and China's forestry industry standards. Secondly, mobile phones were utilized to capture the olive images at the different maturity periods under natural conditions. An olive image dataset was then constructed. A series of operations were implemented to adjust the contrast, the image size, salt-pepper noise, and artificial occlusion, in order to expand the olive images to four times the original ones. The training and test set was built at the ratio of 9:1. Thirdly, an improved EfficientDet was proposed to detect the ripeness of olive fruits, the inconspicuous changes of the adjacent ripeness features, and the mutual occlusion between the fruits. The main advantages of this model were as follows: 1) A module of convolution block attention was introduced into the feature extraction network of the EfficientDet model. The weight distribution of features was then calculated in the channel and spatial domain. The improved model was then used to accurately locate the target area, thereby eliminating the interference of background information. The phenotypic characteristics of olive fruit were fully extracted for the key features to distinguish maturity categories. 2) A cross-level data flow was added to fuse the features of the bottom nodes into the high-level nodes for common learning. As such, the weighted bidirectional feature pyramid network of the EfficientDet model fully utilized the semantic relationship and location information of the high-level and bottom layers. Finally, the improved EfficientDet model was verified with the 623 olive test images. The experimental results showed that the precision, recall, and mean average precision of the improved EfficientDet model under the test set were 92.89%, 93.59%, and 94.60%, respectively, where the average detection time was only 0.337s, and the model size was only 32.4M. The mean average precision was improved by 4.77 percentage points, but the average detection time and model size were reduced by 0.028 s and 14 M, respectively, compared with the original. The mean average precision of the improved EfficientDet model increased by 7.85, 3.73, and 1.04 percentage points, respectively, compared with the SSD, YOLOv3, and Faster R-CNN model. Correspondingly, the model sizes were reduced by 60.8, 202.6, and 75.6 M, respectively. The mean average precision increased by 1.15 percentage points, while the model size increased by 4.9M, compared with the YOLOv5s model. Therefore, the improved EfficientDet model presented a higher detection accuracy with a smaller memory size, fully meeting the actual needs of promotion and application under natural conditions.