Abstract: In the visual system of picking robot, the recognition and orientation of fruit is the key technology. In the natural environment, there is complex light and the shadows are ubiquitous physical phenomena in the natural world. The light would be blocked by branches, leaves, fruit, etc., resulting in shadow casting on fruits. Shadow makes it more difficult for machine vision to identify and locate fruit, so it is significant to detect and remove shadows in the application of picking robot. In this paper, the shadow detection and removal method for fruit recognition by picking robot in the natural environment was studied to avoid the effect of light changes during the day, and the experiments were designed to verify the feasibility and effectiveness of the algorithm. In this study, we used bounding box manually to circumscribe and tag the branches, leaves, fruit, ground and sky of the citrus, litchi and longana images. The classification labels of the bounding box were set to shadow and no shadow firstly. The shadow region and the no shadow region of the orchard image under natural light were compared and analyzed. According to the characteristics of shadows, 8 regional features, including average pixel value of regional grayscale, normalization of the feature, regional feature based on MSRCR transformation, regional feature based on MSRCR transformation, regional feature based on MSRCR transformation, reegional feature based on MSRCR transformation, regional feature based on MSRCR transformation and regional feature based on MSRCR transformation were studied and proved to be the effective features for shadow detection. The 8 self-explored custom features were extracted based on MSRCR and the classification labels of the above regions and trained by using the SVM. K-fold cross validation method was used to optimize the parameters of the SVM, and finally the optimal classification model was obtained. Secondly, the method of superpixel segmentation was used to divide an image into multiple small regions. Based on the superpixel segmentation of the image, the 8 self-explored custom features were extracted, and each small segment of the superpixel segmentation in the image was detected, and it was determined whether each small region was in the shadow. According to the strategy of Finlayson's two-dimensional integration algorithm, the shadow removal was performed on each detected shadow region, and the natural light image was obtained after removal of the shadow. Finally, the accuracy of shadow detection was tested. The experimental results showed that the average accuracy of the shadow detection algorithm in this study was 83.16%. In order to verify the effect of litchi recognition after shadow removal, some methods, including Otsu, K-means and FCM, were implemented to recognize litchis using Cr component of YCbCr color model. The results showed that the litchis were recognized more intactly from the images after shadow removal, shadow removal can improve the recognition effect of fruits. This study provides a technical support for the robots to identify fruits and other industrial and agricultural application scenarios in the natural environment.