Fusion of millimeter wave radar and machine vision for visual guide line extraction on field roads

Field roads have been the most important transportation ways for commercial grain and agricultural products in hilly and mountainous regions in China. Especially, the youth labor can rapidly migrate from the rural to the urban areas against ever-increasing urbanization. Current manual production cannot fully meet the large-scale and precision agriculture in recent years, owing to the protracted yield cycle and low rate of return. Therefore, it is urgent to implement the mechanization and intelligent agriculture for the national food security in hilly and mountainous regions. Among them, the autonomous and safe operation of intelligent machinery can be critical to navigate or circumvent obstacles on field roads. However, the conventional machine vision cannot accurately and rapidly construct the visual guidance lines in such terrains. In this study, the extraction was proposed to enhance the visual guidance and obstacle avoidance using millimeter-wave radar and vision fusion. The state information was also detected from the target objects on the field road. The specific steps were as follow. Preprocessing techniques were employed to filter the portion of the radar object data. A multi-target tracking was utilized to eliminate the interference data for the continuous tracking of dynamic objects. Accurate radar object data was then obtained for subsequent data fusion. A semantic segmentation network was created using Deeplabv3+, and then leveraged a dataset of the adjacent field roads. The millimeter-wave radar and vision data were synchronized in both time and space via the timestamp alignment and least squares-based coordinate transformation. Extraction approach was then established for the visual guidance lines, particularly for the scenarios where dynamic target state was both available and unavailable. A series of experiments were carried out to validate the extraction of visual guide line. The average errors of detection were ranged from 1.60 to 9.20 pixels at the real road midpoints in the scenes without dynamic targets. Moreover, the visual guidance lines were successfully extracted for the obstacle avoidance proactively when encountering dynamic targets. Evidently, the integrated approach was effectively overcome the constraints of traditional machine vision, thereby enhancing the safety and reliability of machinery operation in hilly and mountainous terrains. Meanwhile, the efficacy was depended mainly on the precise projection of millimeter-wave radar data onto the visual plane. Simultaneously, the variables were markedly introduced the positional discrepancies, such as the posture of platform (including pitch and roll). Furthermore, the driveability of the operational area was effectively evaluated in the forthcoming regions. At the same time, the real-world conditions were more complex in the frequent presence of common elements like livestock, vehicles, and tricycles. The resilience of approach can be expected to improve the radar data processing for the detection of static objects. Moreover, positional deviations can also be mitigated from mechanical movements. As such, the precise visual guidance lines were established in the hilly and mountainous terrains. Therefore, the radar data processing can be focused on the dataset expanding and position deviation that induced by various factors. The intelligent level of agricultural machinery can be improved in hilly and mountainous area. The finding is of great significance to consolidate national food security.