Detection of ridge in front of vehicle based on fusion of camera and millimeter wave radar

Detection of the front ridge is an important step for navigation and path planning of autonomous agricultural vehicles. In single sensor detection, it is hard to acquire enough information, such as shape, distance, and height, because of the complex environment in the field. In this study, the novel detection of the front ridge was proposed to integrate the camera and millimetre-wave radar. The camera was used to collect the shape, while the radar was used to collect the distance and height of the front ridge. More detailed information of the front ridge was achieved after fusion of the data acquired by the camera and millimetre-wave radar. In visual detection, the distribution characteristics of the front ridge were used in the images, while the gradient resampling was used to accelerate image processing. Only less than 1% of the total needed to be processed. A support vector machine (SVM) was then applied with 11-dimensional colour-texture features in image segmentation. The 11-dimensional colour-texture features contained three-dimensional colour features in RGB colour space, four-dimensional colour features in HSI colour space, and four-dimensional texture features in gray level co-occurrence matrix, indicating both colour and texture features of the front ridge. Furthermore, the equal-width hypothesis of the geometric feature was used to obtain a more accurate shape of the front ridge. The equal-width hypothesis referred to that there were no sharp curvature changes of the front ridge in the images. Some misjudgement points were filtered in this hypothesis. Millimetre-wave radar was installed vertically in the millimetre-wave radar detection. Compared with the common horizontal one, the vertical installation was used to ensure the installation height and bumpy ground, while the height of ridge at the same time. In fusion detection, the millimetre-wave radar data was transferred to the image coordinate system via coordinate transformation formula and pinhole imaging model. The visual detection was then used to filter interference points in the millimetre-wave radar data. An accurate distance of the front ridge was captured, and the interference points in the millimetre-wave radar data were filtered easily using the coupled camera and millimetre-wave radar. Both camera and millimetre-wave radar were two-dimensional sensors, but after fusion, the three-dimensional information was achieved, like shape, distance, and height of the front ridge. The camera and millimetre-wave radar enhanced each other. The radar was placed at different heights and angles in both horizontal and vertical installation, in order to verify the vertical placement of radar. Tests showed that the horizontal installation was greatly affected by the installation height and terrain turbulence, but the vertical installation effectively overcame these effects. A dataset was recorded to verify the correctness of fusion, including 300 images and 50 groups of fusion data with different distance and shooting angles of the front ridge. The test was performed on the Nvidia Jetson TX2 hardware platform, where the visual detection spent 40.83 ms per image, and the accuracy was 95.67%, the average angle deviation was 0.67°, the average offset deviation was 2.69%. The accuracy was slightly reduced by 1.33%, the average angle deviation was slightly reduced by 0.04°, the average offset deviation was slightly reduced by 0.14 percentage points, but the detection speed was improved by 794.11 ms, compared with the traditional whole image processing. The average deviation of distance detection was 0.11 m in fusion detection, the standard deviation of distance detection was 6.93 cm, and the average deviation of height detection was 0.13 m. Consequently, the standard deviation of height detection was 0.19 m. The fusion detection of the camera and millimetre-wave radar can meet the requirements of real-time and accuracy for autonomous agricultural vehicles.