Article Details

Title · Authors · Abstract · Keywords · DOI · References

Multi-modal Fusion 3D Object Detection Method Based on PointPillars

WANG Ruizhi, ZHANG Hong

(Taiyuan Normal University, Jinzhong 030619, China)

Section：Artificial Intelligence · Vol. 2025, Issue 5 · Pages：56-61 · 2096-4706（2025）05-0056-06 · CLC number：TP183；TP391 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2025.05.010

Abstract

The sparsity and disorder of point cloud data lead to the problems of missed detection and false detection in the detection of distant and small objects. Therefore, this paper proposes a multi-modal fusion 3D object detection algorithm based on the PointPillars algorithm. The algorithm designs a multi-modal feature fusion pillar encoding module, which can fuse point cloud features and image features, thereby enhancing the semantic information of features and improving the detection accuracy of distant and small objects. The experimental results on the KITTI dataset show that compared with the baseline model, the 3D average detection accuracy of the vehicle, pedestrian and cyclist categories is improved by 3.28%, 2.88% and 1.62%, respectively. The results show that the proposed multi-modal fusion 3D object detection method based on PointPillars can effectively reduce the false detection and missed detection of distant and small objects.

Keywords

multi-modal fusion; 3D Object Detection; pillar encoding; PointPillars

References

  1. [1] QIAN R，LAI X，LI X R. 3D Object Detection for Autonomous Driving: A Survey [J/OL].Pattern Recognition，2022，130：108796[2024-08-26].https://doi.org/10.1016/j.patcog.2022.108796.
  2. [2] MA X Z，OUYANG W L，SIMONELLI A，et al. 3D Object Detection From Images for Autonomous Driving：A Survey [J].IEEE Transactions On Pattern Analysis And Machine Intelligence，2024，46（5）：3537-3556.
  3. [3] WANG Y J，MAO Q Y，ZHU H Q，et al. Multi-Modal 3D Object Detection in Autonomous Driving: A Survey [J].International Journal of Computer Vision，2023，131（8）：2122-2152.
  4. [4] LANG A H，VORA S，CAESAR H，et al. PointPillars: Fast Encoders for Object Detection From Point Clouds [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach：IEEE，2019：12689-12697.
  5. [5] HUANG K L，SHI B T，LI X，et al. Multi-modal Sensor Fusion for Auto Driving Perception: A Survey [J/OL].arXiv:2202.02703 [cs.CV].[2024-08-03].https://doi.org/10.48550/arXiv.2202.02703.
  6. [6] VORA S，LANG A H，HELOU B，et al. PointPainting: Sequential Fusion for 3D Object Detection [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：4604-4612.
  7. [7] CHEN X Z，MA H M，WAN J，et al. Multi-View 3D Object Detection Network for Autonomous Driving [J/OL].arXiv:1611.07759 [cs.CV].[2024-08-15].https://doi.org/10.48550/arXiv.1611.07759.
  8. [8] KU J，MOZIFIAN M，LEE J，et al. Joint 3D Proposal Generation and Object Detection from View Aggregation [C]//2018 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS).Madrid：IEEE，2018：1-8.
  9. [9] XU D F，ANGUELOV D，LAIN A. PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：244-253.
  10. [10] HUANG T T，LIU Z，CHEN X W，et al. EPNet: Enhancing Point Features with Image Semantics for 3D Object Detection [C]//Computer Vision-ECCV 2020.Glasgow：Springer，2020：35-52.
  11. [11] PAIGWAR A，SIERRA-GONZALEZ D，ERKENT O，et al. Frustum-PointPillars: A Multi-Stage Approach for 3D Object Detection using RGB Camera and LiDAR [C]//.2021 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW).Montreal：IEEE，2021：2926-2933.
  12. [12] WANG K，ZHOU T Q，ZHANG Z C，et al. PVF-DectNet: Multi-modal 3D Detection Network Based on Perspective-Voxel Fusion [J/OL].Engineering Applications of Artificial Intelligence，2023，120：105951[2024-07-26].https://doi.org/10.1016/j.engappai.2023.105951.
  13. [13] CHEN Z H，LI Z Y，ZHANG S Q，et al. AutoAlign: Pixel-Instance Feature Aggregation for Multi-Modal 3D Object Detection [J/OL].arXiv:2202.02703 [cs.CV].[2024-08-11].https://doi.org/10.48550/arXiv.2202.02703.
  14. [14] ZHOU Y，TUZEL O. VoxelNet: End-to-End Learning for Point Cloud Based 3D Object Detection [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：4490-4499.
  15. [15] YAN Y，MAO Y X，LI B. SECOND:Sparsely Embedded Convolutional Detection [J/OL]. Sensors，2018，18（10）：3337[2024-07-23].https://doi.org/10.3390/s18103337.
  16. [16] LIU Z，ZHAO X，HUANG T T，et al. TANet: Robust 3D Object Detection from Point Clouds with Triple Attention [C]//Proceedings of the AAAI Conference on Artificial Intelligence.New York：AAAI，2020，34（7）：11677-11684.
  17. [17] PAN X R，XIA Z F，SONG S J，et al. 3D Object Detection with Pointformer [C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville：IEEE，2021：7459-7468.
  18. [18] SHI W J，RAJKUMAR R. Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloud [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：1708-1716.
  19. [19] HE C H，ZENG H，HUANG J Q，et al. Structure Aware Single-Stage 3D Object Detection From Point Cloud [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：11870–11879.

Funding

Graduate Education Innovation Project of Taiyuan Normal University (SYYJSYC-2395)

About the authors

Wang Ruizhi (1999-), male, Han ethnicity, from Yuncheng, Shanxi Province, currently pursuing a master's degree in 3D object detection and image processing; Corresponding author: Zhang Hong (1977-), female, Han ethnicity, from Taiyuan, Shanxi, associate professor, PhD, research direction: artificial intelligence, blockchain, and intelligent data.