Article Details

Title · Authors · Abstract · Keywords · DOI · References

Multimodal Recipe Retrieval Based on Consistency Loss

OUYANG Nanxin, ZHU Xinghui

(College of Information and Intelligence, Hunan Agricultural University, Changsha 410128, China)

Section：Computer Technology · Vol. 2025, Issue 9 · Pages：74-78,83 · 2096-4706（2025）09-0074-06 · CLC number：TP391.4 · Document code：A · DOI：10.19850/j.cnki.2096-4706.2025.09.014

Abstract

Multimodal recipe retrieval effectively matches food images with their corresponding recipe texts. However, semantic inconsistencies between image and text modalities pose challenges to retrieval accuracy and efficiency. This paper proposes a Consistent Multimodal Hierarchical Transformers (CMHT) model, which enhances semantic consistency between food images and recipe texts in the embedding space through cross-modal and intra-modal contrastive learning. The experimental results show that the application of CMHT on the recipe dataset improves the accuracy of retrieval, proving the application potential of this method in multimodal data processing in the food field.

Keywords

intelligent agriculture; food computing; multimodal recipe retrieval; cross-modal contrastive learning; semantic consistency

References

  1. [1] SALVADOR A，HYNES，AYTAR Y，et al. Learning Cross-Modal Embeddings for Cooking Recipes and Food Images [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu：IEEE，2017：3068-3076.
  2. [2] CHEN J J，NGO C W，FENG F L，et al.Deep Understanding of Cooking Procedure for Cross-modal Recipe Retrieval [C]//Proceedings of the 26th ACM International Conference on Multimedia.Seoul：ACM，2018：1020-1028.
  3. [3] CARVALHO M，CADENE R，PICARD D，et al. Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings [C]//The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.Ann Arbor：ACM，2018：35-44.
  4. [4] ZHU B，NGO C W，CHEN J J，et al. R2GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach：IEEE，2019：11469-11478.
  5. [5] WANG H，SAHOO D，LIU C H，et al. Learning Cross-Modal Embeddings With Adversarial Networks for Cooking Recipes and Food Images [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach：IEEE，2020：11564-11573.
  6. [6] FU H，WU R，LIU C H，et al，MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：14558-14568.
  7. [7] ZAN Z C，LI L，LIU J Q，et al. Sentence-based and Noise-robust Cross-modal Retrieval on Cooking Recipes and Food Images [C]//Proceedings of the 2020 International Conference on Multimedia Retrieval.Dublin：ACM，2020：117-125.
  8. [8] SUGIYAMA Y，YANAI K. Cross-Modal Recipe Embeddings by Disentangling Recipe Contents and Dish Styles [C]//Proceedings of the 29th ACM International Conference on Multimedia.[S.I.]：ACM，2021：2501-2509.
  9. [9] PHAM H X，GUERRERO R，LI J T，et al. CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval [J/OL].arXiv:2102.02547 [cs.CV].[2021-09-27].https://doi.org/10.48550/arXiv.2102.02547.
  10. [10] LI J，SUN J L，XU X，et al. Cross-Modal Image-Recipe Retrieval via Intra- and Inter-Modality Hybrid Fusion [C]//Proceedings of the 2021 International Conference on Multimedia Retrieval.Taipei：ACM，2021:173-182.
  11. [11] XIE Z W，LIU L，LI L，et al. Efficient Deep Feature Calibration for Cross-Modal Joint Embedding Learning [C]//Proceedings of the 2021 International Conference on Multimodal Interaction.Montréal：ACM，2021：43-51.
  12. [12] XIE Z W，LIU L，LI L，et al. Learning Joint Embedding with Modality Alignments for Cross-Modal Retrieval of Recipes and Food Images [C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.[S.I.]：ACM，2021：2221-2230.
  13. [13] XIE Z W，LIU L，WU Y Z，et al. Learning Text-image Joint Embedding for Efficient Cross-modal Retrieval with Deep Feature Engineering [J].ACM Transactions on Information Systems (TOIS)，2021，40（4）：1-27.
  14. [14] XIE Z W，LIU L，WU Y Z，et al. Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Service [J].IEEE Transactions on Services Computing，2021，15（6）：3304-3316.
  15. [15] CHEN Y，ZHOU D，LI L，et al. Multimodal Encoders for Food-Oriented Cross-Modal Retrieval [C]//5th International Joint Conference, APWeb-WAIM 2021.Guangzhou：Springer，2021：253-266.
  16. [16] LI L，LI M，ZAN Z C，et al. Multi-subspace Implicit Alignment for Cross-modal Retrieval on Cooking Recipes and Food Images [C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.[S.I.]：ACM，2021：3211-3215.
  17. [17] SALVADOR A，GUNDOGDU E，BAZZANI L，et al. Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning [C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville：IEEE，2021：15470-15479.

About the authors

Ouyang Nanxin (1998-), female, Han ethnicity, from Hengyang, Hunan, currently pursuing a master's degree in agricultural information technology research; Corresponding author: Zhu Xinghui (1971-), male, Han ethnicity, from Loudi, Hunan, professor, doctoral supervisor, PhD, research direction: research on distributed systems and advanced database technology.