Article Details

Abstract

To address the problems of insufficient semantic alignment between text and image as well as difficulty in dynamically balancing modal contributions in text-image multimodal sentiment analysis, this paper proposes a sentiment analysis model integrating gated self-attention, bidirectional inter-modal co-attention and adaptive gated fusion (Multimodal Gated Attention for Cross-modal Learning Network, MGACL-Net). RoBERTa and ResNet101 are employed to extract text and image features, respectively. Intra-modal gated self-attention is utilized to suppress noise, and bidirectional co-attention is adopted to model text-image interaction. Finally, gated fusion is applied to complete sentiment classification. Comparative experiments and ablation experiments are conducted on the MVSA-Single and MVSA-Multiple datasets. The results show that the proposed model achieves an accuracy of 77.12% and an F1 score of 76.92% on the MVSA-Single dataset. The proposed model can strengthen the representation of text-image emotional cues and improve the effectiveness and robustness of cross-modal fusion.

Keywords

multimodal sentiment analysis; text-image fusion; cross-modal attention; adaptive gating; sentiment classification

References

1. [1] Gandhi A，Adhvaryu K，Poria S，et al. MMultimodal Sentiment Analysis: A Systematic Review of History, Datasets, Multimodal Fusion Methods, Applications, Challenges and Future Directions [J].Information Fusion，2023，91：424-444.
2. [2] Zhang Yazhou, Rong Lu, Song Dawei, etc A review of multimodal sentiment analysis research [J]. Pattern Recognition and Artificial Intelligence, 2020, 33 (5): 426-438
3. [3] Zhao K，Zheng M S，Li Q G，et al. Multimodal Sentiment Analysis—A Comprehensive Survey From a Fusion Methods Perspective [J].IEEE Access，2025：64556-64583.
4. [4] Yang Yufei, Qian Yurong, Gong Weijun, etc BERT Image Text Multimodal Sentiment Analysis with Large Model Optimization [J]. Computer System Applications, 2025, 34 (8): 62-69
5. [5] Hu H R. A Vision-Language Pre-training model based on Cross Attention for Multimodal Aspect-based Sentiment Analysis [C]//2024 5th International Conference on Computer Vision, Image and Deep Learning (CVIDL)，2024：370-375.
6. [6] Li S T，Tang H. Multimodal Alignment and Fusion: A Survey [J/OL].arXiv:2411.17040 [cs.CV].https://doi.org/10.48550/arXiv.2411.17040.
7. [7] Zhu Z Y，Jin Z B，Zhang J Y，et al. Narrowing Information Bottleneck Theory for Multimodal Image-Text Representations Interpretability [J/OL].arXiv:2502.14889 [cs.CV].https://doi.org/10.48550/arXiv.2502.14889.
8. [8] Huang Z C，Zeng Z Y，Liu B，et al. Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers [J/OL].arXiv:2004.00849 [cs.CV].https://doi.org/10.48550/arXiv.2004.00849.
9. [9] Radford A，Kim J W，Hallacy C，et al. Learning Transferable Visual Models From Natural Language Supervision [C]//Proceedings of the 38th International Conference on Machine Learning，2021：8748-8763.
10. [10] Liang X，Yang E，Deng C，et al. CrossFormer: Cross-Modal Representation Learning via Heterogeneous Graph Transformer [J].ACM Transactions on Multimedia Computing, Communications and Applications，2024，20（12）：1-21.
11. [11] Feng X Y，Lin Y M，He L H，et al. Knowledge-Guided Dynamic Modality Attention Fusion Framework for Multimodal Sentiment Analysis [J/OL].arXiv:2410.04491 [cs.CL].https://doi.org/10.48550/arXiv.2410.04491.
12. [12] LI H C，LU Y T，ZHU H D. Multi-Modal Sentiment Analysis Based on Image and Text Fusion Based on Cross-Attention Mechanism [J/OL].Electronics，2024，13（11）：2069.https://doi.org/10.3390/electronics13112069.
13. [13] Zhang Yu, Zhang Haijun, Liu Yaqing, etc Multimodal sentiment analysis based on bidirectional masked attention mechanism [J]. Data Analysis and Knowledge Discovery, 2023, 7 (4): 46-55
14. [14] Lv Xueqiang, Tian Chi, Zhang Le, etc A multimodal sentiment analysis model that integrates multiple features and attention mechanisms [J]. Data Analysis and Knowledge Discovery, 2024, 8 (5): 91-101

About the authors

Song Xiwen (2001-), female, Han ethnicity, from Xi'an, Shaanxi Province, is currently pursuing a master's degree in natural language processing and emotion classification.