Abstract:
Effective safe navigation in demanding maritime environments demands dependable obstacle detection at all ranges and in all weather and lighting conditions. Existing mono-modal systems, whether radar, LiDAR or camera, suffer from signifcant drawbacks and cannot be relied upon to operate independently. This project presents a multi-modal sensor fusion software system developed to mitigate these drawbacks and merge the information produced by camera, LiDAR, and radar sensors into a single, comprehensive 360-degree situational awareness image. The system was developed using publicly available maritime dataset: the Pohang Canal dataset. Four independent deep learning models were trained: YOLOv8 for stereo camera object detection, achieving a mAP@50 of 0.917; Deeplab V3 for IR camera; a PointNet binary classifer for LiDAR cluster fltering, achieving 100% accuracy; and a PointNet binary classifer for radar cluster fltering, achieving 82% overall accuracy. We used Depth Anything V2 for monocular depth estimation. It converts camera detections into metric range estimates. The outputs from the model are merged through a fve-stage late-fusion process that systematically addresses stereo camera duplicates, cross-verifes camera detections with infrared readings, enhances detections using LiDAR metric depth, incorporates radar validation, and ensures comprehensive lateral coverage. In a sample evaluation frame, the combined pipeline generated 51 confrmed detections ranging from 21.7 to 1493 meters, signifcantly surpassing the capabilities of any individual sensor. The fndings are displayed via an interactive six-view Matplotlib dashboard specifcally designed for bridge environments
