Article Details

Abstract

In order to improve the accuracy of the image-text matching task, an image-text retrieval method based on Chinese-CLIP model and Prompt mechanism is proposed. On the one hand, the text data is preprocessed. After removing stop words and punctuation marks, the BERT model is used to extract text features. On the other hand, the Convolutional Neural Network is used to extract image features, and the obtained text and image features are serialized to achieve multi-modal feature fusion. When training the model, the Chinese-CLIP large model is used for preliminary training, and then the Prompt mechanism is introduced to fine-tune the model. The experimental results show that the proposed method effectively improves the accuracy and recall rate in the two tasks of text-to-image and image-to-text.

Keywords

image-text retrieval; multi-modal feature fusion; Chinese-CLIP model; Prompt mechanism

References

1. [1] Liu Ying, Guo Yingying, Fang Jie, etc A review of cross modal image and text retrieval research in deep learning [J]. Computer Science and Exploration, 2022, 16 (3): 489-511
2. [2] Zhang Hucheng, Li Leixiao, Liu Dongjiang A review of multimodal data fusion research [J]. Computer Science and Exploration, 2024, 18 (10): 2501-2520
3. [3] Zhao Shilu, Zhang Qiang Frontiers in Deep Learning Multimodal Image Semantic Segmentation [J]. Chinese Journal of Image and Graphics, 2023, 28 (11): 3320-3341
4. [4] Zeng Mingrui, Yuan Mengqi, Shao Xi, etc Research progress on text feature extraction [J]. Journal of Nanjing University of Information Science and Technology: Natural Science Edition, 2019, 11 (6): 706-715
5. [5] Li Weiwei, Wang Liyan, Fu Bo, etc Deep neural network image restoration method based on multimodal fusion [J]. Journal of Jilin University: Science Edition, 2024, 62 (2): 391-398
6. [6] Zhou Luqing, Jia Ke, Feng Ao, etc Integrating self attention mechanism to improve ResNet's image classification method [J]. Software Guide, 2024, 23 (10): 173-178
7. [7] Cao Xuan, Xu Qinglin Emotion analysis integrating attention and comparative reinforcement learning mechanisms [J]. Software Guide, 2022, 21 (1): 124-130
8. [8] Zou Zi'an, He Ruhan Cost sensitive fine-grained clothing image retrieval [J]. Software Guide, 2023, 22 (10): 185-190
9. [9] Chen Xiaojiang, Yang Xiaoqi, Chen Guanghao, etc Low time complexity short text classification using a combination of BERT and width learning [J]. Journal of Shandong University: Engineering, 2024, 54 (4): 51-58+66
10. [10] Li Yang, Wu Xiaoqun A review of multi-layer feature extraction algorithms for RGB-D images [J]. Chinese Journal of Image and Graphics, 2024, 29 (5): 1346-1363

Funding

Guangxi Natural Science Foundation Project (2019GXNSFBA245056); College Student Innovation and Entrepreneurship Training Program Project (20241059673230410596731)

About the authors

Chen Daobin (2003-), male, Han ethnicity, from Fuzhou, Fujian Province, undergraduate studies, research direction: computer vision; Zhang Zinuo (2004-), female, Han ethnicity, from Shijiazhuang, Hebei Province, undergraduate studies, research direction: fractional order differential equations; Fu Yubin (2003-), male, Zhuang ethnicity, from Hechi, Guangxi. He is currently pursuing a bachelor's degree with a research focus on computer vision; Li Jinming (2003-), male, Yao ethnic group, from Hezhou, Guangxi, undergraduate student, research direction: computer vision; Corresponding author: Lin Bin (1985-), male, Han ethnicity, from Dalian, Liaoning, associate professor, PhD, research direction: pattern recognition and machine learning.