Article Details

Title · Authors · Abstract · Keywords · DOI · References

Research on the Intelligent Agent for Locomotive Operation Environment Perception Based on the Multi-modal Large Model

CHEN Zhe, ZHENG Chen, YAN Han, LI Kaixuan, LIU Zheng

(CRSC Communication and Information Group Co., Ltd., Beijing 100071, China)

Section：Computer Technology · Vol. 2026, Issue 5 · Pages：106-111 · 2096-4706（2026）05-0106-06 · CLC number：U291.1；TP183 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2026.05.020

Abstract

Against the problems of foreign Object Detection and risk recognition in rail transit scenarios, this paper proposes an environment perception intelligent agent system based on the multi-modal large model. The system adopts independent detection pathways for vision and millimeter-wave radar, and on the basis of maintaining the complementary advantages of sensors, introduces a Large Language Model intelligent agent module with memory, reasoning and tool-calling capabilities to perform semantic fusion and risk analysis on heterogeneous perception results, thereby realizing a more accurate and flexible abnormal event recognition and response mechanism. Experimental results show that the system has significant advantages in small target recognition, long-range detection and semantic judgment in complex scenarios, and can provide a system paradigm for high-reliability risk early warning in rail transit scenarios.

Keywords

multi-modal fusion; radar-vision collaboration; environment perception; intelligent agent; track foreign object detection

References

  1. [1] ZHANG Q Y，WEI Y，HAN Z，et al. Multimodal Fusion on Low-quality Data: A Comprehensive Survey [J/OL].arXiv:2404.18947，2024-04-27.https://arxiv.org/abs/2404.18947.
  2. [2] LEI L，ZHOU Y，YANG G. Multisource Information Fusion-based Environment Perception and Dynamic Model of Underwater Vehicle in Irregular Ocean Environment [J].Information Fusion，2023，94：257-271.
  3. [3] ASAAD R R，SAEED V A，ABDULHAKIM R M. Smart Agent and It's Effect on Artificial Intelligence: A Review Study [J].Icontech International Journal，2021，5（4）：1-9.
  4. [4] ZHANG C，LIU L，CUI Y，et al. A Comprehensive Survey on Segment Anything Model for Vision and Beyond [J/OL].arXiv:2305.08196 [cs.CV].(2023-05-19).https://arxiv.org/abs/2305.08196.
  5. [5] CHENG T，SONG L，GE Y. Yolo-world: Real-time Open-vocabulary Object Detection [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Piscataway：IEEE，2024：16901-16911.
  6. [6] LIU S，ZENG Z，REN T，et al. Grounding DINO: Marrying DINO with Grounded Pre-training for Open-set Object Detection [C]//European Conference on Computer Vision.Cham: Springer Nature Switzerland，2024：38-55.
  7. [7] SINGH A. Vision-RADAR Fusion for Robotics BEV Detections: A Survey [C]//Proceedings of the 2023 IEEE Intelligent Vehicles Symposium（IV）.Anchorage：IEEE，2023：1-7.
  8. [8] SCHULHOFF S，ILIE M，BALEUP N，et al. The Prompt Report: A Systematic Survey of Prompt Engineering Techniques [J/OL].arXiv:2406.06608 [cs.CL].(2025-02-26).https://arxiv.org/abs/2406.06608.
  9. [9] ZHAO Z，LIU Y，WU H，et al. Clip in Medical Imaging: A Comprehensive Survey [J/OL].arXiv:2312.07353 [cs.CV].(2025-03-26).https://arxiv.org/abs/2312.07353.
  10. [10] ZHU S，LI C，ROGERS J，et al. A Real-time Double Emulsion Droplets Detection System using Hough Circle Transform and Color Detection [C]//Proceedings of the 2021 27th International Conference on Mechatronics and Machine Vision in Practice (M2VIP).New York：IEEE，2021：36-41.

Funding

Beijing Science and Technology Plan Project (Z231100003823031)

About the authors

Chen Zhe (1994-), male, Han ethnicity, from Anyang, Henan Province, China, engineer, master's student, research direction: artificial intelligence.