文章详情

题名 · 作者 · 摘要 · 关键词 · DOI · 参考文献

基于一致性损失的多模态食谱检索

欧阳楠心，朱幸辉

（湖南农业大学 信息与智能科学技术学院，湖南 长沙 410128)

栏目：计算机技术 · 第 2025 年第 9 期 · 页码：74-78,83 · 2096-4706（2025）09-0074-06 · 中图分类号：TP391.4 · 文献标识码：A · DOI：10.19850/j.cnki.2096-4706.2025.09.014

摘要

多模态食谱检索能够有效匹配食物图像与相应的食谱文本。然而，图像和文本模态之间的语义不一致性，给检索精度和效率带来了挑战。文章提出了一种一致性多模态层次Transformer（Consistent Multimodal Hierarchical Transformers, CMHT）模型，通过跨模态与模态内的对比学习，增强了食物图像和食谱文本在嵌入空间中的语义一致性。实验结果表明，CMHT在食谱数据集上的应用提升了检索的准确性，证明了该方法在食品领域多模态数据处理中的应用潜力。

关键词

智能农业；食品计算；多模态食谱检索；跨模态对比学习；语义一致性

参考文献

  1. [1] SALVADOR A，HYNES，AYTAR Y，et al. Learning Cross-Modal Embeddings for Cooking Recipes and Food Images [C]//2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR).Honolulu：IEEE，2017：3068-3076.
  2. [2] CHEN J J，NGO C W，FENG F L，et al.Deep Understanding of Cooking Procedure for Cross-modal Recipe Retrieval [C]//Proceedings of the 26th ACM International Conference on Multimedia.Seoul：ACM，2018：1020-1028.
  3. [3] CARVALHO M，CADENE R，PICARD D，et al. Cross-Modal Retrieval in the Cooking Context: Learning Semantic Text-Image Embeddings [C]//The 41st International ACM SIGIR Conference on Research & Development in Information Retrieval.Ann Arbor：ACM，2018：35-44.
  4. [4] ZHU B，NGO C W，CHEN J J，et al. R2GAN: Cross-Modal Recipe Retrieval With Generative Adversarial Network [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach：IEEE，2019：11469-11478.
  5. [5] WANG H，SAHOO D，LIU C H，et al. Learning Cross-Modal Embeddings With Adversarial Networks for Cooking Recipes and Food Images [C]//2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Long Beach：IEEE，2020：11564-11573.
  6. [6] FU H，WU R，LIU C H，et al，MCEN: Bridging Cross-Modal Gap between Cooking Recipes and Dish Images with Latent Variable Model [C]//2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Seattle：IEEE，2020：14558-14568.
  7. [7] ZAN Z C，LI L，LIU J Q，et al. Sentence-based and Noise-robust Cross-modal Retrieval on Cooking Recipes and Food Images [C]//Proceedings of the 2020 International Conference on Multimedia Retrieval.Dublin：ACM，2020：117-125.
  8. [8] SUGIYAMA Y，YANAI K. Cross-Modal Recipe Embeddings by Disentangling Recipe Contents and Dish Styles [C]//Proceedings of the 29th ACM International Conference on Multimedia.[S.I.]：ACM，2021：2501-2509.
  9. [9] PHAM H X，GUERRERO R，LI J T，et al. CHEF: Cross-modal Hierarchical Embeddings for Food Domain Retrieval [J/OL].arXiv:2102.02547 [cs.CV].[2021-09-27].https://doi.org/10.48550/arXiv.2102.02547.
  10. [10] LI J，SUN J L，XU X，et al. Cross-Modal Image-Recipe Retrieval via Intra- and Inter-Modality Hybrid Fusion [C]//Proceedings of the 2021 International Conference on Multimedia Retrieval.Taipei：ACM，2021:173-182.
  11. [11] XIE Z W，LIU L，LI L，et al. Efficient Deep Feature Calibration for Cross-Modal Joint Embedding Learning [C]//Proceedings of the 2021 International Conference on Multimodal Interaction.Montréal：ACM，2021：43-51.
  12. [12] XIE Z W，LIU L，LI L，et al. Learning Joint Embedding with Modality Alignments for Cross-Modal Retrieval of Recipes and Food Images [C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.[S.I.]：ACM，2021：2221-2230.
  13. [13] XIE Z W，LIU L，WU Y Z，et al. Learning Text-image Joint Embedding for Efficient Cross-modal Retrieval with Deep Feature Engineering [J].ACM Transactions on Information Systems (TOIS)，2021，40（4）：1-27.
  14. [14] XIE Z W，LIU L，WU Y Z，et al. Learning TFIDF Enhanced Joint Embedding for Recipe-Image Cross-Modal Retrieval Service [J].IEEE Transactions on Services Computing，2021，15（6）：3304-3316.
  15. [15] CHEN Y，ZHOU D，LI L，et al. Multimodal Encoders for Food-Oriented Cross-Modal Retrieval [C]//5th International Joint Conference, APWeb-WAIM 2021.Guangzhou：Springer，2021：253-266.
  16. [16] LI L，LI M，ZAN Z C，et al. Multi-subspace Implicit Alignment for Cross-modal Retrieval on Cooking Recipes and Food Images [C]//Proceedings of the 30th ACM International Conference on Information & Knowledge Management.[S.I.]：ACM，2021：3211-3215.
  17. [17] SALVADOR A，GUNDOGDU E，BAZZANI L，et al. Revamping Cross-Modal Recipe Retrieval with Hierarchical Transformers and Self-supervised Learning [C]//2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR).Nashville：IEEE，2021：15470-15479.

作者简介

欧阳楠心（1998—），女，汉族，湖南衡阳人，硕士在读，研究方向：农业信息技术研究；通信作者：朱幸辉（1971—），男，汉族，湖南娄底人，教授，博士生导师，博士，研究方向：分布式系统及高级数据库技术研究。