Heterologous sources images in the apple orchard registration method using EM-PCNN

Image registration has been a key preprocessing technique for heterogeneous images. Two kinds of image data were transformed into the same coordinate system by matching, combining spatial information and semantic information. Multi-source images can be gained to reduce the limitations of a single data source. This study aims to improve the matching accuracy between Time of Flight (ToF) and visible heterologous images collected by the binocular acquisition system in an orchard environment. A target salient region extraction was proposed using the local peak, pulse coupled neural network (pCNN) segmentation and the expectation-maximization for the ToF and visible image registration in an apple orchard. Firstly, the significant region in the visible image was calculated to preprocess the red and green components of the visible image, according to the Gaussian difference function. Taking the two-dimensional normal distribution of the local gray value in the image as the target component, the maximum interclass variance Otsu was used to extract the prospect with a fixed threshold as the local peak extraction strategy, then to preliminarily screen the characteristic areas of ToF and visible images, in order to improve the pCNN dynamic threshold using the maximum expectation. The link strength was calculated using the local image gradient. An Expectation-Maximization (EM) pCNN was proposed to refine the pre-selected region using the image region variance for a better termination condition. The invariant moment of the connected region was calculated to locate the same name point of the target center using the invariant principle, where the characteristic region was further screened. Finally, the same name points were purified by the Random Sample Consensus (RANSAC), where the purified coordinates of the same name points were substituted into the transformation model to calculate the model parameters for the registration. The experimental images were collected under three conditions, including a sunny day, shade, and weak light, in order to simulate different lighting conditions. The translation, spatial rotation, and scaling were performed on each group of ToF and visible images. The experimental results show that the EM-PCNN was better than the traditional segmentation for the image with less obvious color difference between fruit and growth background. Under normal conditions, the segmentation accuracy and the exposure were 96.62% and 73.84% lower, respectively. After that, the feature regions were screened to perform fine segmentation using EM-PCNN. There were smaller differences in regional features between the ToF and visible light, compared with the traditional Maximally Stable Extremal Regions (MSER), indicating a more accurate range. Harris algorithm was suitable for the image registration with the small scale change and rotation angle. The scale-invariant feature transform (Sift) algorithm presented the resistance to the translation, rotation, and scaling, where the complete registration was realized with the rotation at the spatial level. Consequently, the local peak extraction and EM-PCNN segmentation can be widely expected to register the ToF and visible images in the apple orchard, where the root mean square error was 3.05-4.75 under different lighting conditions, and the homonymous point was 3-5. The excellent registration was achieved for the ToF and visible heterologous images in an apple orchard under the binocular acquisition system, indicating better resistance to the image translation, rotation, and scaling.