文章详情

题名 · 作者 · 摘要 · 关键词 · DOI · 参考文献

基于文本提示的多模态图像跟踪系统

张凡，雷江波，李亚学，郭轩

（西安导引科技有限责任公司，陕西 西安 710075）

栏目：人工智能 · 第 2026 年第 9 期 · 页码：57-63 · 2096-4706（2026）09-0057-07 · 中图分类号：TP391 · 文献标识码：A · DOI：10.19850/j.cnki.2096-4706.2026.09.011

摘要

针对纯视觉跟踪器易受遮挡、尺度突变、背景杂乱等干扰因素导致跟踪稳定性差、抗干扰能力弱的问题，本文提出一种多模态融合跟踪方法。引入文本提示思想并构建文本提示库，提出一种视觉与语义信息相融合的多模态跟踪策略。在检测阶段对视频帧候选目标进行细粒度识别，提取目标属性特征以构建文本提示库；将文本提示特征输入视觉孪生网络完成融合，实现文本关键语义信息与视觉区域的关联匹配。实验结果表明，该方法在多个开源数据集上综合性能更优，复杂实景下抗干扰能力更强，跟踪稳定性更高。

关键词

文本提示；多模态融合；细粒度识别；目标跟踪

参考文献

  1. [1] KRISTAN M，MATAS J，LEONARDIS A，et al. A Novel Performance Evaluation Methodology for Single-Target Trackers [J].IEEE transactions on pattern analysis and machine intelligence，2016，38（11）：2137-2155.
  2. [2] REDMON J，DIVVALA S，GIRSHICK R，et al. You Only Look Once: Unified， Real-Time Object Detection [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Las Vegas：IEEE，2016：779-788.
  3. [3] REN S Q，HE K M，GIRSHICK R，et al. Faster R-CNN: Towards Real-time Object Detection with region Proposal Networks [J].IEEE Transactions on Pattern Analysis and Machine Intelligence，2016，39（6）：1137-1149.
  4. [4] LIU W，ANGUELOV D，ERHAN D，et al. SSd: Single Shot Multibox Detector [C]//European conference on computer vision.Cham：Springer International Publishing，2016：21-37.
  5. [5] RADFORD A，KIM J W，HALLACY C，et al. Learning Transferable Visual Models From Natural Language Supervision [J/OL].arXiv:2103.00020 [cs.CV].[2025-11-19].https://doi.org/10.48550/arXiv.2103.00020.
  6. [6] LI C L，LIANG X Y，LU Y J，et al. RGB-T Object Tracking: Benchmark and Baseline [J/OL].Pattern Recognition，2019，96：106977[2025-11-19].https://doi.org/10.1016/j.patcog.2019.106977.
  7. [7] YAN S，YANG J Y，KÄPYLÄ J，et al. Depthtrack: Unveiling the Power of Rgbd Tracking [C]//Proceedings of the IEEE/CVF International Conference on Computer Vision.Montreal：IEEE，2021：10705-10713.
  8. [8] WANG X，LI J N，ZHU L，et al. Visevent: Reliable Object Tracking Via Collaboration of Frame and Event Flows [J].IEEE Transactions on Cybernetics，2023，54（3）：1997-2010.
  9. [9] TANG Z Y，XU T Y，ZHU X F，et al. Omni Survey for Multimodality Analysis in Visual Object Tracking [EB/OL].[2025-11-18].https://arxiv.org/html/2508.13000v1.
  10. [10] LI Z Y，TAO R，GAVVES E，et al. Tracking by Natural Language Specification [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Honolulu：IEEE，2017：7350-7358.
  11. [11] WANG X，SHU X J，ZHANG Z P，et al. Towards More Flexible and Accurate Object Tracking with Natural Language: Algorithms and Benchmark [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and pattern Recognition.Nashville：IEEE，2021：13763-13773.
  12. [12] 莫智懿.基于多模态信息融合的目标跟踪算法研究 [D].桂林：广西师范大学，2024.
  13. [13] FENG Q，ABLAVSKY V，BAI Q X，et al. Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers [C]//Proceedings of the IEEE/CVF conference on computer vision and pattern recognition.Nashville：IEEE，2021：5847-5856.
  14. [14] SUN Y M，YU F，CHEN S X，et al. Chattracker: Enhancing Visual Tracking Performance Via Chatting with Multimodal Large Language Model [J/OL].arXiv:2411.01756 [cs.CV].[2026-11-18].https://doi.org/10.48550/arXiv.2411.01756.
  15. [15] LI X，HUANG Y Q，HE Z Y，et al. Citetracker: Correlating Image and Text for Visual Tracking [C]//Proceedings of the IEEE/CVF international conference on computer vision.Pari：IEEE，2023：9940-9949.
  16. [16] LIU P F，YUAN W Z，FU J L，et al. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing [J].ACM computing surveys，2023，55（9）：1-35.
  17. [17] WANG Z F，ZHANG Z Z，LEE C Y，et al. Learning to Prompt for Continual Learning [C]//2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition.New Orleans：IEEE，2022：139-149.
  18. [18] LEE D J，SONG S，SUH J，et al. Read-Only Prompt Optimization for Vision-Language Few-shot Learning [C]//Proceedings of the IEEE/CVF international conference on computer vision.Paris：IEEE，2023：1401-1411.
  19. [19] DENG J，DONG W，SOCHER R，et al. Imagenet: A Large-scale Hierarchical Image Database [C]//2009 IEEE Conference on Computer Vision and Pattern Recognition.Miami：IEEE，2009：248-255.
  20. [20] LI B，WU W，WANG Q，et al. Siamrpn++:Evolution of Siamese Visual Tracking with Very Deep Networks [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：4282-4291.
  21. [21] SHALEV-SHWARTZ S. Online Learning and Online Convex Optimization [J].Foundations and Trends® in Machine Learning，2012，4（2）：107-194.
  22. [22] GIRSHICK R. Fast R-CNN [J/OL].arXiv:1504.08083 [cs.CV].[2026-11-18].https://doi.org/10.48550/arXiv.1504.08083.
  23. [23] FAN H，LIN L T，YANG F，et al. LaSOT: A High-Quality Benchmark for Large-Scale Single Object Tracking [C]//2019 IEEE/CVF conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2019：5369-5378.
  24. [24] LI Z Y，TAO R，GAVVES E，et al. Tracking by Natural Language Specification [C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.Long Beach：IEEE，2017：7350-7358.
  25. [25] BERTINETTO L，VALMADRE J，HENRIQUES J F，et al. Fully-Convolutional Siamese Networks for Object Tracking [C]//European Conference on Computer Vision.Cham：Springer International Publishing，2016：850-865.
  26. [26] GUO M Z，ZHANG Z P，FAN H，et al. Divert More Attention to Vision-Language Tracking [J/OL].arXiv:2207.01076 [cs.CV].[2026-11-19].https://doi.org/10.48550/arXiv.2207.01076.
  27. [27] FENG Q，ABLAVSKY V，BAI Q X，et al. Siamese Natural Language Tracker: Tracking by Natural Language Descriptions with Siamese Trackers [C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition.Nashville：IEEE，2021：5847-5856.
  28. [28] ZHENG Y Z，ZHONG B N，LIANG Q H，et al. Toward Unified Token Learning for Vision-Language Tracking [J].IEEE Transactions on Circuits and Systems for Video Technology，2023，34（4）：2125-2135.

作者简介

张凡（1995.01—），男，汉族，陕西西安人，助理工程师，硕士，研究方向：智能优化算法；雷江波（1996.07—），男，汉族，陕西渭南人，工程师，硕士，研究方向：图像处理；李亚学（1993.08—），男，汉族，陕西渭南人，工程师，硕士，研究方向：计算机视觉；郭轩（1989.09—），男，汉族，陕西渭南人，工程师，硕士，研究方向：计算机视觉。