Article Details

Abstract

Current image-text cross-modal retrieval has two main bottlenecks, including the fact that traditional Attention Mechanism often includes a large number of redundant regions and introduces irrelevant semantic noise, and that excessive screening leads to insufficient effective regions and causes the loss of key visual information. These two situations significantly reduce the matching accuracy and robustness of the model. To address this problem, this paper proposes a dual optimization strategy. It first adaptively retains high-response regions through a dual filtering mechanism to effectively suppress redundant noise. Meanwhile, it innovatively introduces a Top-K dynamic completion method to automatically supplement key semantic regions when feature deficiency is detected. Experimental verification shows that this method effectively avoids the loss of key information while maintaining the accuracy of feature selection. It significantly improves the cross-modal matching performance of the model in complex scenes.

Keywords

cross-modal retrieval; image-text retrieval; feature alignment; threshold filtering; attention optimization

References

1. [1] HARDOON D R，SZEDMAK S，SHAWE-TAYLOR J. Canonical Correlation Analysis: An Overview with Application to Learning Methods Unavailable [J].Neural Computation，2004，16（12）：2639-2664.
2. [2] ZHENG W M，ZHOU X Y，ZOU C R，et al. Facial Expression Recognition Using Kernel Canonical Correlation Analysis (KCCA) [J].IEEE Transactions on Neural Networks，2006，17（1）：233-238.
3. [3] BENTON A，KHAYRALLAH H，GUJRAL B，et al. Deep Generalized Canonical Correlation Analysis [C]//Proceedings of the 4th Workshop on Representation Learning for NLP （RepL4NLP-2019）.Florence：ACL，2019：1-6.
4. [4] LEE K-H，CHEN X，HUA G，et al. Stacked Cross Attention for Image-Text Matching [C]//Proceedings of the European Conference on Computer Vision (ECCV).Munich：Springer，2018：212-228.
5. [5] LU J S，BATRA D，PARIKH D，et al. ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks [J/OL].arXiv:1908.02265 [cs.CV].[2025-06-01].https://doi.org/10.48550/arXiv.1908.02265.
6. [6] TAN H，BANSAL M. LXMERT: Learning Cross-Modality Encoder Representations from Transformers [C]//Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).Hong Kong：ACL，2019：5100-5111.
7. [7] RADFORD A，KIM J W，HALLACY C，et al. Learning Transferable Visual Models From Natural Language Supervision [C]//Proceedings of the International Conference on Machine Learning (ICML).[S.I.]：PMLR，2021，139：8748-8763.
8. [8] JIA C，YANG Y F，XIA Y，et al. Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision [J/OL].arXiv:2102.05918 [cs.CV].[2025-05-27].https://doi.org/10.48550/arXiv.2102.05918.
9. [9] FAGHRI F，FLEET D J，KIROS J R，et al. VSE++: Improving Visual-Semantic Embeddings with Hard Negatives [J/OL].arXiv:1707.05612 [cs.LG].[2025-06-05].https://doi.org/10.48550/arXiv.1707.05612.
10. [10] WANG H R，ZHANG Y，JI Z，et al. Consensus-Aware Visual-Semantic Embedding for Image-Text Matching [C]//Computer Vision - ECCV 2020.Glasgow：Springer，2020：18-34.
11. [11] KRISHNA R，ZHU Y，GROTH O，et al. Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations [J].International Journal of Computer Vision，2017，123（1）：32-73.
12. [12] REN S Q，HE K M，GIRSHICK R，et al. Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks [J/OL].arXiv:1506.01497 [cs.CV].[2025-07-03].https://doi.org/10.48550/arXiv.1506.01497.
13. [13] ANDERSON P，HE X D，BUEHLER C，et al. Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answering [C]//2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Salt Lake City：IEEE，2018：6077-6086.
14. [14] CHO K，MERRIËNBOER B V，GULCEHRE C，et al. Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation [J/OL].arXiv:1406.1078 [cs.CL].[2025-06-02].https://doi.org/10.48550/arXiv.1406.1078.
15. [15] PENNINGTON J，SOCHER R，MANNING C D. GloVe: Global Vectors for Word Representation [C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP).Doha：ACL，2014：1532-1543.
16. [16] YOUNG P，LAI A，HODOSH M，et al. From Image Descriptions to Visual Denotations: New Similarity Metrics for Semantic Inference over Event Descriptions [J].Transactions of the Association for Computational Linguistics，2014，2：67-78.
17. [17] LIN T Y，MAIRE M，BELONGIE S，et al. Microsoft COCO: Common Objects in Context [C]//Proceedings of the European Conference on Computer Vision.Zurich：Springer，2014：740-755.
18. [18] LIU C X，MAO Z D，LIU A A，et al. Focus Your Attention: A Bidirectional Focal Attention Network for Image-Text Matching [C]//Proceedings of the 27th ACM International Conference on Multimedia. New York：ACM，2019：3-11.
19. [19] LI K P，ZHANG Y L，LI K，et al. Visual Semantic Reasoning for Image-Text Matching [C]//2019 IEEE/CVF International Conference on Computer Vision (ICCV).Seoul：IEEE，2019：4653-4661.
20. [20] CHEN H，DING G G，LIU X D，et al. IMRAM: Iterative Matching With Recurrent Attention Memory for Cross-Modal Image-Text Retrieval [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：12652-12660.

Funding

Natural Science Foundation of Jilin Province (20230101242JC)

About the authors

Meng Fanqi (1981-), male, Mongolian ethnicity, from Kulun Banner, Inner Mongolia, professor, doctoral degree, research interests include computer science and technology; Corresponding author: Tian Kaidi (1999-), female, Han ethnicity, from Jilin, Jilin, master's student, research interests include image processing and cross-modal learning; Tian Yan (1999-), male, Han ethnicity, from Changchun, Jilin, master's student, research interests include image processing.