Article Details

Abstract

Scene graphs represent images as structured semantic graphs, where nodes denote objects and their attributes and edges denote relationships between objects, thereby enriching visual information representation. Research on scene graph generation shows remarkable progress, but question answering based on scene graphs still requires further exploration. To this end, this study constructs a Bilinear Bipartite Graph Attention Network (BB-GAN) framework that selects the most relevant nodes from the scene graph based on the question content. First, the question is encoded into a fully connected graph, and a graph attention network is used to iteratively update the information. Subsequently, a bipartite graph between the semantic structure graph and the scene graph is constructed, BB-GAN is used for node matching, and a modular attention mechanism is introduced to enhance modality alignment. Global scene features, keyword node features, and question features are adaptively fused for answer prediction. Experiments on the GQA dataset demonstrate that the proposed method significantly outperforms existing approaches on binary questions. Ablation studies further validate the effectiveness of BB-GAN.

Keywords

scene graph question answering; bilinear bipartite graph attention network; graph attention network; modality alignment

References

1. [1] Gao Yalong, Zhu Jinlong Visual question answering method based on multi granularity hierarchical knowledge [J]. Communication and Information Technology, 2025 (S1): 45-48
2. [2] Jiang Limei, Li Binglong A review of multimodal processing methods for image text [J]. Computer Application Research, 2024, 41 (5): 1281-1290
3. [3] Kong Fanyan, Liu Lijun, Zhang Yunfeng, etc Medical visual question answering method based on semantic information consistency verification [J]. Chinese Journal of Information Science, 2025, 39 (10): 143-155
4. [4] Chen Qiaohong, Xiang Shenxiang, Fang Xian, etc Cross modal adaptive feature fusion based visual question answering method [J]. Journal of Harbin Institute of Technology, 2025, 57 (4): 94-104
5. [5] Zhang Haitao, Guo Xinyu A scene text visual question answering model based on multimodal inference graph neural network [J]. Computer Application Research, 2022, 39 (1): 280-284302
6. [6] Senior H，Slabaugh G，Yuan S X，et al. Graph neural networks in vision-language image understanding: a survey [J].The Visual Computer，2024，41（1）：1-2.
7. [7] Liang W X，Jiang Y H，Liu Z X. GraphVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answering [J/OL].Computing Research Repository，2021，abs/2104.10283.https://doi.org/10.48550/arXiv.2104.10283.
8. [8] Li Linhao, Wang Yize, Li Yingshuang, etc A panoramic scene image generation method based on relationship feature enhancement [J]. Computer Applications, 2025, 45 (2): 584-593
9. [9] Jung N G，Chang C. Union-Redefined Prototype Network for scene graph generation [J/OL].Expert Systems With Applications，2025，280：127486.https://doi.org/10.1016/j.eswa.2025.127486.
10. [10] Jia D D，Zhou M L，Wei W，et al. A Novel Two-Stage Training Method for Unbiased Scene Graph Generation via Distribution Alignment [J].KSII Transactions on Internet & Information Systems，2023，17（12）：3383-3397.
11. [11] Talupuri P K，Kumar M G，Lavanya K，et al. Smart MCQ Generation using KeyBERT and BERT Based Models [J].2024 3rd Edition of IEEE Delhi Section Flagship Conference (DELCON)，2024:1-5.
12. [12] Teney D，Liu L，Hengel A V D. Graph-Structured Representations for Visual Question Answering [C]//Computer Vision and Pattern Recognition，2017：3233-3241.
13. [13] Hudson D A，Manning C D. GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answering [J]. 019 IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR)，2019：6693-6702.
14. [14] Pennington J，Socher R，Manning C. GloVe: Global Vectors for Word Representation [J].Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing(EMNLP)，2014：1532-1543.
15. [15] Kato J，Mita A，Gobara K，et al. Deep Graph Attention Networks [C]//2024 Twelfth International Symposium on Computing and Networking Workshops(CANDARW)，2024：170-174.
16. [16] Carion N，Massa F，Synnaeve G，et al. End-to-End Object Detection with Transformers [J].Lecture Notes in Computer Science，2020：213-229.
17. [17] Lulidan, Xia Haiying, Tan Yumei, etc Attention guided joint learning of local features for facial expression recognition [J]. Chinese Journal of Image and Graphics, 2024, 29 (8): 2377-2387

About the authors

Jia Dongdong (1996-), male, Han nationality, from Yan'an, Shaanxi, teaching assistant, master's student, research direction: computer science, image processing, scene understanding and perception; Xue Yuxin (1996-), female, Han ethnicity, from Yan'an, Shaanxi, teaching assistant, master's student, research direction: intelligent control, structural parameter optimization, reliability analysis; Zhang Zhaoxin (2003-), female, Han ethnicity, from Weinan, Shaanxi, currently pursuing a master's degree in computer vision and image processing; Li Tianyu (2002-), male, Han ethnicity, from Xi'an, Shaanxi, is currently pursuing a master's degree in small sample classification of hyperspectral images.