Article Details

Title · Authors · Abstract · Keywords · DOI · References

Text-Prompted Multimodal Image Tracking System

ZHANG Fan, LEI Jiangbo, LI Yaxue, GUO Xuan

(Xi'an Daoyin Technology Co., Ltd., Xi'an 710075, China)

Section：Artificial Intelligence · Vol. 2026, Issue 9 · Pages：57-63 · 2096-4706（2026）09-0057-07 · CLC number：TP391 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2026.09.011

Abstract

Aiming at the problems of poor tracking stability and weak anti-interference ability caused by interference factors such as occlusion, scale changes, and background clutter in pure visual trackers, this paper proposes a multimodal fusion tracking approach. This study introduces the concept of text prompts and constructs a text prompt library, proposing a multimodal tracking strategy that integrates visual and semantic information. Fine-grained detection of candidate targets in video frames is performed during the detection phase, extracting target attribute features to build a text prompt library. The text prompt features are fed into a visual Siamese network for fusion, achieving the association and matching between key semantic information from text prompts and visual regions. Experimental results show that this method achieves better comprehensive performance on multiple open-source datasets, stronger anti-interference ability in complex real-world scenarios, and higher tracking stability.

Keywords

text prompt; multimodal fusion; fine-grained recognition; object tracking

References

  1. [1] KRISTAN M，MATAS J，LEONARDIS A，et al. A Novel Performance Evaluation Methodology for Single-Target Trackers [J].IEEE transactions on pattern analysis and machine intelligence，2016，38（11）：2137-2155.
  2. [2] REDMON J，DIVVALA S，GIRSHICK R，et al. You Only Look Once: Unified， Real-Time Object Detection [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas：IEEE，2016：779-788.
  3. [3] REN S Q，HE K M，GIRSHICK R，et al. Faster R-CNN: Towards Real-time Object Detection with region Proposal Networks [J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2016，39（6）：1137-1149.
  4. [4] LIU W，ANGUELOV D，ERHAN D，et al. SSd: Single Shot Multibox Detector [C]//European conference on computer vision.Cham：Springer International Publishing，2016：21-37.
  5. [5] RADFORD A，KIM J W，HALLACY C，et al. Learning Transferable Visual Models From Natural Language Supervision [J/OL].arXiv:2103.00020 [cs.CV].[2025-11-19].https://doi.org/10.48550/arXiv.2103.00020.
  6. [6] LI C L，LIANG X Y，LU Y J，et al. RGB-T Object Tracking: Benchmark and Baseline [J/OL].Pattern Recognition，2019，96：106977[2025-11-19].https://doi.org/10.1016/j.patcog.2019.106977.
  7. [7] YAN S，YANG J Y，KÄPYLÄ J，et al. Depthtrack: Unveiling the Power of Rgbd Tracking [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal：IEEE，2021：10705-10713.
  8. [8] WANG X，LI J N，ZHU L，et al. Visevent: Reliable Object Tracking Via Collaboration of Frame and Event Flows [J].IEEE Transactions on Cybernetics，2023，54（3）：1997-2010.
  9. [9] TANG Z Y，XU T Y，ZHU X F，et al. Omni Survey for Multimodality Analysis in Visual Object Tracking [EB/OL].[2025-11-18].https://arxiv.org/html/2508.13000v1.
  10. [10] LI Z Y，TAO R，GAVVES E，et al. Tracking by Natural Language Specification [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu：IEEE，2017：7350-7358.
  11. [11] WANG X，SHU X J，ZHANG Z P，et al. Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and pattern Recognition.Nashville：IEEE，2021：13763-13773.
  12. [12] Mo Zhiyi Research on Target Tracking Algorithm Based on Multimodal Information Fusion [D]. Guilin: Guangxi Normal University, 2024
  13. [13] FENG Q，ABLAVSKY V，BAI Q X，et al. Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers [C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.Nashville：IEEE，2021：5847-5856.
  14. [14] SUN Y M，YU F，CHEN S X，et al. Chattracker: Enhancing Visual Tracking Performance Via Chatting with Multimodal Large Language Model [J/OL].arXiv:2411.01756 [cs.CV].[2026-11-18].https://doi.org/10.48550/arXiv.2411.01756.
  15. [15] LI X，HUANG Y Q，HE Z Y，et al. Citetracker: Correlating Image and Text for Visual Tracking [C]//Proceedings of the IEEE/CVF international conference on computer vision.Pari：IEEE，2023：9940-9949.
  16. [16] LIU P F，YUAN W Z，FU J L，et al. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [J].ACM computing surveys，2023，55（9）：1-35.
  17. [17] WANG Z F，ZHANG Z Z，LEE C Y，et al. Learning to Prompt for Continual Learning [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans：IEEE，2022：139-149.
  18. [18] LEE D J，SONG S，SUH J，et al. Read-Only Prompt Optimization for Vision-Language Few-shot Learning [C]//Proceedings of the IEEE/CVF international conference on computer vision.Paris：IEEE，2023：1401-1411.
  19. [19] DENG J，DONG W，SOCHER R，et al. Imagenet: A Large-scale Hierarchical Image Database [C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami：IEEE，2009：248-255.
  20. [20] LI B，WU W，WANG Q，et al. Siamrpn++:Evolution of Siamese Visual Tracking with Very Deep Networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：4282-4291.
  21. [21] SHALEV-SHWARTZ S. Online Learning and Online Convex Optimization [J].Foundations and Trends® in Machine Learning，2012，4（2）：107-194.
  22. [22] GIRSHICK R. Fast R-CNN [J/OL].arXiv:1504.08083 [cs.CV].[2026-11-18].https://doi.org/10.48550/arXiv.1504.08083.
  23. [23] FAN H，LIN L T，YANG F，et al. LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking [C]//2019 IEEE/CVF conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：5369-5378.
  24. [24] LI Z Y，TAO R，GAVVES E，et al. Tracking by Natural Language Specification [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2017：7350-7358.
  25. [25] BERTINETTO L，VALMADRE J，HENRIQUES J F，et al. Fully-Convolutional Siamese Networks for Object Tracking [C]//European Conference on Computer Vision.Cham：Springer International Publishing，2016：850-865.
  26. [26] GUO M Z，ZHANG Z P，FAN H，et al. Divert More Attention to Vision-Language Tracking [J/OL].arXiv:2207.01076 [cs.CV].[2026-11-19].https://doi.org/10.48550/arXiv.2207.01076.
  27. [27] FENG Q，ABLAVSKY V，BAI Q X，et al. Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville：IEEE，2021：5847-5856.
  28. [28] ZHENG Y Z，ZHONG B N，LIANG Q H，et al. Toward Unified Token Learning for Vision-Language Tracking [J].IEEE Transactions on Circuits and Systems for Video Technology，2023，34（4）：2125-2135.

About the authors

Zhang Fan (1995.01-), male, Han ethnicity, from Xi'an, Shaanxi Province, assistant engineer, master's degree, research direction: intelligent optimization algorithms; Lei Jiangbo (July 1996-), male, Han ethnicity, from Weinan, Shaanxi, engineer, master's degree, research direction: image processing; Li Yaxue (August 1993-), male, Han ethnicity, from Weinan, Shaanxi, engineer, master's degree, research direction: computer vision; Guo Xuan (September 1989-), male, Han ethnicity, from Weinan, Shaanxi, engineer, master's degree, research direction: computer vision.