Abstract: Abstract: Various wheat diseases can seriously deteriorate the quality and yield to decline significantly, thereby to restrict the high-quality and sustainable development of modern agriculture in China. An accurate and efficient identification of wheat scab become urgent to control the spread of pests and diseases, and to guarantee for the wheat yield. In this study, a research object was taken as the image of wheat ears that infected with scab during grouting period. A multi-way convolutional neural network was designed to identify the wheat scab images based on the color distribution characteristics of the diseased and healthy areas on the research object. Firstly, a deep semantic segmentation network U-Net was used to segment the wheat images in the field environment to remove the influence of wheat leaves and other unrelated backgrounds, particularly on densely growing wheat ears, the cluttered backgrounds of wheat leaves and soil, and complex outdoor lighting. Since the segmentation can efficiently reduce the noise from complex backgrounds, the image of a single wheat ear was segmented for the subsequent image recognition of wheat scab. Then, a simple multi-way convolutional neural network was used to extract the feature information via three R,G,B channels of the wheat ear images. Three feature vectors can be output by the convolutional neural network, and then be merged by vector stitching at the end of the network. After the last pooling layer, the feature vectors were selected to be stitched in the channel dimension in order to form thicker features. This processing can enrich the features described by the wheat data sample in a high-level feature vector. Finally, a joint loss function was used to further improve the performance of the network, particularly on the image detection from the infected wheat ears with head blight, in order to reduce the detecting time for the difference between the features within the head of blight mildew images. After the segmentation, 2 745 images of complete single wheat scab were obtained via the comparison of 510 wheat images that collected in the field environment. A multi-way convolutional neural network combined color channels can increase the width of entire network, and enhance the utilization of each layer of channels, indicating that each layer can learn rich features, such as texture features in different directions and different frequencies. The experimental results showed that the joint loss function learning can increase the distance between different classes, whereas reduce the distance between the same class, thereby to make the network extract more robust features for the identification of wheat scab based on the multi-way convolutional neural network. The findings demonstrated that the proposed algorithm can achieve 100% recognition accuracy for wheat scab, and further provide a valuable support for the intelligent identification of wheat diseases.