Article Details

Abstract

With the emergence of multi-source medical data such as high-throughput genome sequencing and high-resolution digital pathological images, multimodal biological modeling becomes the key to artificial intelligence-assisted pathological diagnosis. This study proposes a new multimodal representation learning method, GTransFusion, to jointly analyze pathological Whole Slide Images and omics data, so as to improve the diagnostic accuracy of various cancers. This method maps different modal data into a unified sequence representation through a Transformer-based joint representation learning module, explicitly models modal type encoding in the process, and realizes dynamic modal weighting by virtue of the self-attention mechanism. Meanwhile, this method constructs a cross-modal feature alignment graph structure, utilizes a Graph Neural Network to capture inter-modal association and common information, and feeds back to the Transformer representation learning to realize cross-modal feature alignment and relationship modeling. Experiments on multiple tumor datasets show that the proposed method is significantly superior to comparison methods in survival prediction performance indicators, which verifies the effectiveness of multimodal joint representation and graph structure alignment.

Keywords

multimodal fusion; Transformer; heterogeneous graph; joint representation learning

References

1. [1] BENKIRANE H，VAKALOPOULOU M，PLANCHARD D，et al. Multimodal CustOmics: A Unified and Interpretable Multi-task Deep Learning Framework for Multimodal Integrative Data Analysis in Oncology [J/OL].PLOS Computational Biology，2025，21（6）：e1013012[2026-01-28].https://doi.org/10.1371/journal.pcbi.1013012.
2. [2] STEYAERT S，PIZURICA M，NAGARAJ D，et al. Multimodal Data Fusion for Cancer Biomarker Discovery with Deep Learning [J].Nature Machine Intelligence，2023，5（4）：351-362.
3. [3] LIN X V，SHRIVASTAVA A，LUO L，et al. MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts [J/OL].arXiv:2407.21770 [cs.AI].[2025-06-20].https://doi.org/10.48550/arXiv.2407.21770.
4. [4] YANG Q H，ZHAO Y，CHENG H. MMLF: Multi-modal Multi-class Late Fusion for Object Detection with Uncertainty Estimation [J/OL].arXiv:2410.08739 [cs.CV].[2025-06-25].https://doi.org/10.48550/arXiv.2410.08739.
5. [5] BOULAHIA S Y，AMAMRAA，MADI M R，et al. Early, intermediate and late fusion strategies for robust deep learning-based multimodal action recognition [J/OL].2021，32：121[2025-07-06].https://link.springer.com/article/10.1007/s00138-021-01249-8.
6. [6] CHEN R J，LU M Y，WANG J W，et al. Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis [J].IEEE Transactions on Medical Imaging，2022，41（4）：757-770.
7. [7] HONG Y，FANG S H，SU J Q，et al. A Novel Approach for Image Encryption with Chaos-RNA [J].Computers, Materials & Continua，2023，77（1）：139-160.
8. [8] DOSOVITSKIY A，BEYER L，KOLESNIKOV A，et al. An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale [J/OL].arXiv:2010.11929 [cs.CV].[2025-06-17].https://doi.org/10.48550/arXiv.2010.11929.
9. [9] SCHNEIDER L，LAIOUAR-PEDARI S，KUNTZ S，et al. Integration of Deep Learning-Based Image Analysis and Genomic Data in Cancer Pathology: A Systematic Review [J].European Journal of Cancer，2022，160：80-91.
10. [10] LUO H，HUANG J S，JU H R，et al. Multimodal Multi-instance Evidence Fusion Neural Networks for Cancer Survival Prediction [J/OL].Scientific Reports，2025，15：10470[2026-01-28].https://doi.org/10.1038/s41598-025-93770-3.
11. [11] DING K X，ZHOU M，METAXAS D N，et al. Pathology-and-Genomics Multimodal Transformer for Survival Outcome Prediction [C]//Medical Image Computing and Computer Assisted Intervention-MICCAI 2023.Vancouver：Springer，2023：619-629.
12. [12] LI C Y，ZHU X L，YAO J W，et al. Hierarchical Transformer for Survival Prediction Using Multimodality Whole Slide Images and Genomics [C]//2022 26th International Conference on Pattern Recognition (ICPR).Montreal：IEEE，2022：4256-4262.
13. [13] LIU M X，LIU Y Z，CUI H，et al. MGCT: Mutual-Guided Cross-Modality Transformer for Survival Outcome Prediction Using Integrative Histopathology-Genomic Features [C]//2023 IEEE International Conference on Bioinformatics and Biomedicine (BIBM).Istanbul：IEEE，2023：1306-1312.
14. [14] CHEN R J，LU M Y，WILLIAMSON D F K，et al. Pan-cancer Integrative Histology-Genomic Analysis via Multimodal Deep Learning [J].Cancer Cell，2022，40（8）：865-878.
15. [15] LU M Y，WILLIAMSON D F K，CHEN T Y，et al. Data-efficient and Weakly Supervised Computational Pathology on Whole-slide Images [J].Nature Biomedical Engineering，2021，5：555-570.
16. [16] VELIKOVIĆ P，CUCURULL G，CASANOVA A，et al. Graph Attention Networks [J/OL].arXiv:1710.10903 [stat.ML].[2025-06-19].https://arxiv.org/abs/1710.10903.
17. [17] DEVLIN J，CHANG M W，LEE K，et al. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding [J/OL].arXiv:1810.04805 [cs.CL].[2025-06-27].https://arxiv.org/abs/1810.04805.
18. [18] SUBRAMANIAN A，TAMAYO P，MOOTHA V K，et al. Gene Set Enrichment Analysis: A Knowledge-Based Approach for Interpreting Genome-Wide Expression Profiles [J].PNAS，2005，102（43）：15545-15550.

About the authors

Zhang Xian (1999-), male, Han ethnicity, from Zaozhuang, Shandong, is currently pursuing a master's degree in medical image processing and multimodal fusion; Corresponding author: Pang Hui (1979-), female, Han ethnicity, from Zhangjiakou, Hebei Province. She is a master's supervisor, professor, and master's student, with research interests in graph neural networks, blockchain consensus algorithms, etc; Liu Jiajun (2000-), male, Han ethnicity, from Qinhuangdao, Hebei Province, is currently pursuing a master's degree in graph neural networks.