Recognition of maize and weed based on multi-scale hierarchical features extracted by convolutional neural network

Abstract: Effective recognition method of crop and weed is the basis for promoting the development of intelligent mechanization weeding pattern. Summarizing the previous research, we found that the accuracy and stability of the recognition model could be improved by natural and diversified feature presentation, but there are still 2 main problems. On the one hand, feature presentation of the natural property of target was difficult to be obtained by the hand-engineered feature extractor. The spatial consistency of the obtained features was bad, and the real-time performance of recognition system was reduced for the complex feature extraction algorithm. On the other hand, the effect of image preprocessing has important influence on recognition results, especially the overlapping segmentation of crop and weed. For overlapped objects, it is usually difficult to segment them without affecting their respective feature presentations, resulting in low recognition accuracy and stability. In order to solve the main problems in the current research, we explored the way to improve the recognition accuracy, stability and real-time performance, and a recognition method of crop and weed based on multiscale hierarchical feature learning combined with superpixels segmentation was proposed. The main research contents of this paper were as follows: 1) Excellent internal features of image are hierarchical. In this research, the multiscale hierarchical feature is a scene level feature with invariance and consistency in scale space. Multiscale convolutional neural network was built to extract multiscale hierarchical feature. Multiscale convolutional neural network contains multiple copies of a single CNN (convolutional neural network) that are applied to multi-scale Gaussian pyramid of the input image. The CNN model as feature extractor in this paper includes 3 stages. In the first 2 stages, it contains a bank of filters (convolution kernels) to produce dense feature maps, a point to point nonlinear mapping activation function, and a spatial pooling module for sub-sampling of each feature map, While the last stage only contains a bank of filters. Each filter is applied to the input feature maps through 2-dimensional convolution operation, in which local feature presentations are detected at all pixel locations on the input image. For each pixel, the CNN model is used to collectively encode the internal information in a large sense region around the location of given pixel. The CNN is fed with raw pixels and trained with back propagation method. With complete training, this CNN model can automatically extract hierarchical feature representations from the input image, thereby decreasing the need for hand-engineered features extracting. A series of feature maps for multiscale regions centered on each pixel of the image are produced by the multiscale convolutional neural network; these representations contain shape, texture and sense information. Therefore, the multiscale hierarchical feature is learned to allow the recognition of the class of all pixels in the image. The average pixel recognition rate is 93.41%. 2) In this research, multiscale hierarchical feature is used to produce the class distribution for every pixel through a 2-layer MLP (multi-layer perceptron). But recognizing the class of pixel independently from its surrounding regions could produce some interference at the boundary of targets. Accurate boundary segmentation for each target in the image is not provided. In this paper, the superpixel method is used for generating an over segmentation of the image, where each segmentation component (superpixel) is an irregular pixel block composed of similar pixels in texture, color and brightness characteristics. Object in the image is composed of these superpixels, and has the exact original boundary. The superpixels segmentation is computed following the simple linear iterative clustering method described in this paper. 3) We proposed the recognition strategy of multiscale hierarchical feature learning combined with the superpixels segmentation. Firstly, an over segmentation of the original image was produced through the superpixels segmentation. At the same time, each pixel location of the image was classified densely based on the multiscale hierarchical features. These predictions of pixels in each superpixel were aggregated to produce the class prediction of superpixel, through computing the average class distribution within the superpixel. Adjacent superpixels with the same class were merged to obtain the final target class prediction and image segmentation. The accurate image segmentation was achieved while recognizing the target in the image by this recognition method, which effectively avoided the problems caused by targets overlapping, and the recognition results were more stable and accurate. Maize seedlings at 2-5 leaves stage and weed during the same stage were used as research object, and the recognition method of multiscale hierarchical feature learning combined with superpixels segmentation was tested. The results showed that the average target recognition rate with this method was 98.92%, and the standard deviation was 0.55%. The average target recognition rate with the method in previous research was 98.36%, and the standard deviation was 1.05%. So, the accuracy and stability of recognition results by this method in this paper were improved with different degrees. In aspect of real-time performance, the average time to recognize a single image was 1.68 s with this method, and compared with the method proposed in previous research, it was decreased by 1.58 s. The real-time performance of this method can be further improved by GPU (graphics processing unit) hardware acceleration, and the average time to recognize a single image was only 0.72 s. Therefore, the recognition method based on multiscale hierarchical feature learning combined with superpixels segmentation can effectively achieve accurate, stable and efficient recognition of maize and weed. The research results provide reference for the development of precision weeding.