Article Details

Abstract

To address the issues that single 2D images are susceptible to light and perspective interference in the detection methods of highway pavement diseases, and that although 3D point cloud data can represent the three-dimensional morphology of diseases, they have high feature dimensions and high computational costs, a multimodal feature fusion highway pavement disease detection network model based on DETR is proposed. Firstly, a multi-scale feature extraction module and a spatial attention module are integrated into the DETR network structure to construct the MSF-DETR detection network model. Then, on the basis of MSF-DETR, the MFFM multimodal feature fusion module is integrated, and the AWCM adaptive weight calculation module is introduced for the fusion of feature maps of 2D and 3D point cloud data. Finally, through the RSRD dataset, ablation experiments are conducted, achieving the mean Average Precision (mAP) of 93.43%. The mAP of the multimodal feature fusion highway pavement disease detection network model based on DETR is significantly better than that of the single detection methods based only on 2D images or only on 3D point cloud data, effectively achieving the multimodal feature fusion of 2D images and 3D point cloud data, and improving the detection accuracy of highway pavement diseases in complex environments.

Keywords

highway pavement disease detection; 2D image; 3D point cloud; DETR; multi-modal feature fusion

References

1. [1] Gavilán M，Balcones D，Marcos O，et al. Adaptive Road Crack Detection System by Pavement Classification [J].Sensors，2011，11（10）：9628-9657.
2. [2] Shi Y，Cui L M，Qi Z Q，et al. Automatic Road Crack Detection Using Random Structured Forests [J].IEEE Transactions on Intelligent Transportation Systems，2016，17（12）：3434-3445.
3. [3] Hui L，Chen J，Jian L. Road Surface Disease Detection Algorithm Based on Improved YOLOv4 [J/OL].Laser & Optoelectronics Progress，2021，58：1410025.https://m.researching.cn/articles/OJfb397f682b560753/.
4. [4] Wu Y ，Liu Y. Research on Road Surface Defect Detection Method Based on Improved YOLOv8 [J].Automation Application，2025，66（8）：45-49.
5. [5] Pan Ning, Du Yuchuan, Yue Jinsong, etc Automated detection method for road deformation diseases based on point cloud data [J]. Journal of Tongji University (Natural Science Edition), 2022, 50 (3): 399-408
6. [6] Yu Hao Research on Automatic Identification Technology of Road Cracks Based on 3D Laser Point Cloud [D]. Wuhan: Hubei University of Technology, 2015
7. [7] Zhong M Q，Sui L C，Wang Z H，et al. Pavement Crack Detection From Mobile Laser Scanning Point Clouds Using a Time Grid [J].Sensors，2020，20（15）：1-15.
8. [8] Jiang Wanqi Research on Intelligent Identification of Asphalt Pavement Cracks Based on Laser Point Cloud [D]. Nanjing: Southeast University, 2022
9. [9] Liao Jinqiao Road surface damage detection method based on LiDAR point cloud [D]. Beijing: North China University of Technology, 2022
10. [10] Qi C R，Su H，Mo K，et al. PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：652-660.
11. [11] Qi C R，Yi L，Su H，et al. PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Space [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition，2017：5105-5114.
12. [12] Wulff F，Schaufele B，Sawade O，et al. Early Fusion of Camera and Lidar for Robust Road Detection Based on U-Net FCN [C]//2018 IEEE Intelligent Vehicles Symposium (IV)，2018：1426-1431.
13. [13] Elamin A，El-Rabbany A. UAV-Based Image and LiDAR Fusion for Pavement Crack Segmentation [J].Sensors，2023，23（23）：26.
14. [14] Jin Yufeng, Tao Chongben Fusion Information Enhanced 3D Object Detection Algorithm Based on Transformer [J]. Journal of Instrumentation, 2023, 44 (12): 297-306
15. [15] Pang S，Morris D，Radha H. CLOCs: Camera-LiDAR Object Candidates Fusion for 3D Object Detection [C]//IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)，2020：10386-10393.
16. [16] Carion N，Massa F，Synnaeve G，et al. End-to-End Object Detection With Transformers [C]//Computer Vision-ECCV 2020，2020：213-229.
17. [17] Chen Jingyi Research on Tire Surface Defect Detection Algorithm Based on Image and 3D Point Cloud Feature Fusion [D]. Hangzhou: Zhejiang University, 2023
18. [18] Zhao T，Xu C F，Ding M Y，et al. RSRD: A Road Surface Reconstruction Dataset and Benchmark for Safe and Comfortable Autonomous Driving [J].IEEE Transactions on Intelligent Transportation Systems，2024，25（1）：1-15.

Funding

Shanghai Education Development Foundation and Shanghai Education Commission's "Morning Light Plan" project funding (23CGB15)

About the authors

Jia Chen (1996-), female, Han nationality, from Xinxiang, Henan Province, teaching assistant, master's student, research direction: image processing, computer vision.