基于提示调优的少样本多模态关系抽取分层网络

doi:10.19678/j.issn.1000-3428.0070493

计算机工程 ›› 2026, Vol. 52 ›› Issue (7): 368-376. doi: 10.19678/j.issn.1000-3428.0070493

基于提示调优的少样本多模态关系抽取分层网络

李维淦¹^,², 桂宁³, 钱育蓉²^,⁴^,*(), 陈嘉颖¹^,²

1. 新疆大学软件学院, 新疆乌鲁木齐 830046
2. 新疆大学软件工程重点实验室, 新疆乌鲁木齐 830091
3. 中南大学计算机学院, 湖南长沙 410083
4. 新疆大学计算机科学与技术学院, 新疆乌鲁木齐 830091

收稿日期:2024-10-16 修回日期:2025-02-25 出版日期:2026-07-15 发布日期:2026-07-04
通讯作者: 钱育蓉
作者简介:
李维淦(CCF学生会员), 男, 硕士研究生, 主研方向为多模态信息抽取
桂宁, 教授、博士
钱育蓉(CCF高级会员、通信作者), 教授、博士
陈嘉颖, 副教授、博士
基金资助:
国家自然科学基金(62266043); 新疆维吾尔自治区重点研发专项(2023B01029-1); 新疆维吾尔自治区重点研发专项(2023B01029-2); 天山创新团队项目(2023D14012); 新疆维吾尔自治区杰出青年科学基金(2023D01E01); 青年拔尖人才项目(2023TSYCCX0043)

Hierarchical Network for Few-shot Multimodal Relation Extraction Based on Prompt-tuning

LI Weigan¹^,², GUI Ning³, QIAN Yurong²^,⁴^,*(), CHEN Jiaying¹^,²

1. School of Software, Xinjiang University, Urumqi 830046, Xinjiang, China
2. Key Laboratory of Software Engineering, Xinjiang University, Urumqi 830091, Xinjiang, China
3. School of Computer Science and Engineering, Central South University, Changsha 410083, Hunan, China
4. School of Computer Science and Technology, Xinjiang University, Urumqi 830091, Xinjiang, China

Received:2024-10-16 Revised:2025-02-25 Online:2026-07-15 Published:2026-07-04
Contact: QIAN Yurong

摘要/Abstract

摘要：

多模态关系抽取(MRE)方法通过利用多模态信息辅助关系抽取任务。现有MRE模型要达到良好的关系抽取效果, 通常需要在大量标注数据的基础上进行有效训练, 然而在少样本情况下却表现不佳。针对上述问题, 本文充分利用关系标签中丰富的语义知识和先验知识, 提出一种新的提示调优分层网络。首先, 设计实现基于知识注入的文本提示模块, 具体来说, 通过利用关系标签中隐含的实体信息, 引入虚拟实体类型词构建提示模板, 使模型可以感知样本中潜在的实体类型范围, 并利用上下文对引入的虚拟关系答案词进行不断优化, 以表达最佳的语义信息, 从而提升模型在少样本情况下的性能。其次, 利用实体对与关系之间的相互制约关系, 设计实现实体-关系协同优化模块, 以进一步提升关系抽取效果。最后, 在文本编码器的每个自注意力层中, 通过引入基于视觉前缀的注意力机制, 将分层的多尺度视觉特征与文本信息深度融合, 从而生成更加有效且鲁棒的文本表示, 显著降低模型对错误的敏感性。在多模态神经关系抽取数据集(MNRE)上的实验结果表明, 该模型的精确率、召回率、F1值分别达到84.97%、83.91%与84.43%, 较基准模型均有一定提升, 特别是在少样本情况下, 本文所提模型显著优于基准模型, 展现出良好的关系抽取效果。

关键词: 关系抽取, 少样本, 提示调优, 先验知识, 多模态信息

Abstract:

The Multimodal Relation Extraction (MRE) method assists in relation extraction tasks by using multimodal information. To achieve a good relation extraction performance, existing MRE models usually need to be effectively trained on a large amount of labeled data. However, they perform poorly in the case of few-shot samples. To address this issue, this study utilizes the rich semantic and prior knowledge of relation labels and proposes a new prompt-tuning hierarchical network. First, a text-prompt module based on knowledge injection is designed and implemented. Specifically, by utilizing the entity information hidden in relation labels and introducing virtual entity type words to construct prompt templates, the model can perceive the potential range of entity types in the sample and continuously optimize the introduced virtual relation answer words using context to express the best semantic information, thereby improving the performance of the model in the case of few-shot samples. Second, by utilizing the mutual constraint relationship between entity pairs and relations, an entity—relation collaborative optimization module is designed and implemented to further improve the effect of relation extraction. Finally, in each self-attention layer of the text encoder, a visual prefix-based attention mechanism is introduced to deeply integrate the layered multiscale visual features with the text information, thereby generating more effective and robust text representations and significantly reducing the sensitivity of the model to errors. The experimental results on the multimodal neural relation extraction dataset (MNRE) show that the precision, recall, and F1 score of the model reach 84.97%, 83.91%, and 84.43%, respectively, which are all improved compared with the benchmark model. Especially in the case of few-shot samples, the model proposed in this study is significantly better than the benchmark model, demonstrating good relation extraction performance.

Key words: relation extraction, few-shot, prompt-tuning, prior knowledge, multimodal information

李维淦, 桂宁, 钱育蓉, 陈嘉颖. 基于提示调优的少样本多模态关系抽取分层网络[J]. 计算机工程, 2026, 52(7): 368-376.

LI Weigan, GUI Ning, QIAN Yurong, CHEN Jiaying. Hierarchical Network for Few-shot Multimodal Relation Extraction Based on Prompt-tuning[J]. Computer Engineering, 2026, 52(7): 368-376.

https://www.ecice06.com/CN/Y2026/V52/I7/368

图/表 7

图1 提示调优分层网络的模型结构

Fig.1 Structure of prompt-tuning hierarchical network

图2 百分比随机采样的实验结果

Fig.2 The results of experiments randomly sampled by percentage

图3 超参数对比实验结果

Fig.3 The results of the hyperparameter comparison experiment

参考文献 35

1	ZHENG C M, WU Z W, FENG J H, et al. MNRE: a challenge multimodal dataset for neural relation extraction with visual evidence in social media posts[C]//Proceedings of the IEEE International Conference on Multimedia and Expo (ICME). Shenzhen, China: IEEE Press, 2021: 1-6.
2	贺泽华, 夏方毅, 王健, 等. 原型对比学习驱动的鲁棒性关系抽取方法. 重庆邮电大学学报(自然科学版), 2025, 37 (1): 17- 28.
	HE Z H , XIA F Y , WANG J , et al. Robust relation extraction method based on prototypical contrastive learning. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2025, 37 (1): 17- 28.
3	李冬梅, 张扬, 李东远, 等. 实体关系抽取方法研究综述. 计算机研究与发展, 2020, 57 (7): 1424- 1448.
	LI D M , ZHANG Y , LI D Y , et al. Review of entity relation extraction methods. Journal of Computer Research and Development, 2020, 57 (7): 1424- 1448.
4	CALIFF M E, MOONEY R J. Relational learning of pattern-match rules for information extraction[C]//Proceedings of the 16th National Conference on Artificial Intelligence. Berlin, Germany: Springer, 1999: 328: 334.
5	HUFFMAN S B. Learning information extraction patterns from examples[C]//Proceedings of International Joint Conference on Artificial Intelligence. Berlin, Germany: Springer, 1995: 246-260.
6	ZHANG Y H, ZHONG V, CHEN D Q, et al. Position-aware attention and supervised data improve slot filling[C]//Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing. Copenhagen, Denmark: Association for Computational Linguistics, 2017: 35-45.
7	GUO Z J, NAN G S, LU W, et al. Learning latent forests for medical relation extraction[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence. Yokohama, Japan: International Joint Conferences on Artificial Intelligence Organization, 2020: 3651-3657.
8	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[C]// Proceedings of Conference on the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [S. l. ]: Association for Computational Linguistics, 2019: 3498-4195.
9	LI J, WANG R X, ZHANG N Y, et al. Logic-guided semantic representation learning for zero-shot relation classification[C]//Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain: Association for Computational Linguistics, 2020: 2967-2978.
10	ZENG D J, LIU K, CHEN Y B, et al. Distant supervision for relation extraction via piecewise convolutional neural networks[C]//Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics, 2015: 1753-1762.
11	SOARES L B, FITZGERALD N, LING J, et al. Matching the blanks: distributional similarity for relation learning[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, 2019: 2895-2905.
12	CHEN X, ZHANG N Y, XIE X, et al. KnowPrompt: knowledge-aware prompt-tuning with synergistic optimization for relation extraction[C]//Proceedings of the ACM Web Conference 2022. New York, USA: ACM Press, 2022: 2823-2833.
13	ZHENG C M, FENG J H, FU Z, et al. Multimodal relation extraction with efficient graph alignment[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 5298-5306.
14	CHEN X, ZHANG N Y, LI L, et al. Good visual guidance make a better extractor: hierarchical visual prefix for multimodal entity and relation extraction[C]//Proceedings of the Findings of the Association for Computational Linguistics: NAACL, 2022. Seattle, USA: Association for Computational Linguistics, 2022: 1607-1618.
15	吴海鹏, 钱育蓉, 冷洪勇. 基于双向注意力机制的多模态关系抽取. 计算机工程, 2024, 50 (4): 160- 167. doi: 10.19678/j.issn.1000-3428.0067700
	WU H P , QIAN Y R , LENG H Y . Multimodal relation extraction based on bidirectional attention mechanism. Computer Engineering, 2024, 50 (4): 160- 167. doi: 10.19678/j.issn.1000-3428.0067700
16	XU B, HUANG S, DU M, et al. A unified visual prompt tuning framework with mixture-of-experts for multimodal information extraction[C]//Proceedings of International Conference on Database Systems for Advanced Applications. Berlin, Germany: Springer, 2023: 544-554.
17	CUI S Y , CAO J X , CONG X , et al. Enhancing multimodal entity and relation extraction with variational information bottleneck. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, 32, 1274- 1285. doi: 10.1109/TASLP.2023.3345146
18	BROWN T B, MANN B, RYDER N, et al. Language models are few-shot learners[C]//Proceedings of the 34th International Conference on Neural Information Processing Systems. Washington D.C., USA: IEEE Press, 2020: 1877-1901.
19	BEN-DAVID E, OVED N, REICHART R. PADA: example-based prompt learning for on-the-fly adaptation to unseen domains[C]//Proceedings of the Transactions of the Association for Computational Linguistics. [S. l. ]: Association for Computational Linguistics: 414-433.
20	DING N, CHEN Y L, HAN X, et al. Prompt-learning for fine-grained entity typing[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022: 6888-6901.
21	HU S D, DING N, WANG H D, et al. Knowledgeable prompt-tuning: incorporating knowledge into prompt verbalizer for text classification[C]//Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Dublin, Ireland: Association for Computational Linguistics, 2022: 2225-2240.
22	GAO T Y, FISCH A, CHEN D Q. Making pre-trained language models better few-shot learners[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). [S. l. ]: Association for Computational Linguistics, 2021: 3816-3830.
23	SCHICK T, SCHMID H, SCHVTZE H. Automatically identifying words that can serve as labels for few-shot text classification[C]//Proceedings of the 28th International Conference on Computational Linguistics. Barcelona, Spain: Association for Computational Linguistics, 2020: 5569-5578.
24	SHIN T, RAZEGHI Y, LOGAN IV R L, et al. AutoPrompt: eliciting knowledge from language models with automatically generated prompts[C]//Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). [S. l. ]: Association for Computational Linguistics, 2020: 4222-4235.
25	HAMBARDZUMYAN K, KHACHATRIAN H, MAY J. WARP: word-level adversarial reprogramming[C]//Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers). Stroudsburg, USA: Association for Computational Linguistic, 2021: 4921-4933.
26	LIU Z, LIN Y T, CAO Y, et al. Swin Transformer: hierarchical vision transformer using shifted windows[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Montreal, Canada: IEEE Press, 2022: 9992-10002.
27	LIU Y H, OTT M, GOYAL N, et al. RoBERTa: a robustly optimized BERT pretraining approach[EB/OL]. [2024-09-01]. https://arxiv.org/abs/1907.11692.
28	YANG Z Y, GONG B Q, WANG L W, et al. A fast and accurate one-stage approach to visual grounding[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Seoul, Korea: IEEE Press, 2019: 4682-4692.
29	LU D, NEVES L, CARVALHO V, et al. Visual attention model for name tagging in multimodal social media[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Stroudsburg, USA: Association for Computational Linguistic, 2018: 1990-1999.
30	LI J , SUN A X , HAN J L , et al. A survey on deep learning for named entity recognition. IEEE Transactions on Knowledge and Data Engineering, 2022, 34 (1): 50- 70. doi: 10.1109/TKDE.2020.2981314
31	ZHONG Z X, CHEN D Q. A frustratingly easy approach for entity and relation extraction[C]//Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. [S. l. ]: Association for Computational Linguistics, 2021: 50-61.
32	YU J F, JIANG J, YANG L, et al. Improving multimodal named entity recognition via entity span detection with unified multimodal transformer[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. [S. l. ]: Association for Computational Linguistics, 2020: 3342-3352.
33	ZHANG D, WEI S, LI S, et al. Multi-modal graph fusion for named entity recognition with targeted visual guidance[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l. ]: AAAI Press, 2021: 14347-14355.
34	LI L H, YATSKAR M, YIN D, et al. VisualBERT: a simple and performant baseline for vision and language[EB/OL]. [2024-09-01]. https://arxiv.org/abs/1908.03557.
35	HE X Y , LI S X , ZHANG Y N , et al. The more quality information the better: hierarchical generation of multi-evidence alignment and fusion model for multimodal entity and relation extraction. Information Processing & Management, 2025, 62 (1): 103875.

[1]	刘军平, 黄宇威, 胡新荣, 彭涛, 姚迅, 王帮超, 杨华利, 朱强. 基于数据增强和动态阈值的文档级关系抽取方法研究[J]. 计算机工程, 2026, 52(4): 131-139.
[2]	艾传鲜, 郭军军, 尹兆良. 基于层级软提示交互融合的少样本事件方面类别检测方法[J]. 计算机工程, 2025, 51(9): 120-128.
[3]	逯暄, 景路琪, 彭甫镕. 基于增量学习的结直肠息肉分割方法[J]. 计算机工程, 2025, 51(7): 284-293.
[4]	孙丽郡, 孟繁军, 徐行健. 课程知识图谱构建技术研究综述[J]. 计算机工程, 2025, 51(11): 1-21.
[5]	郭桦宜, 游进国, 耿齐祁, 陶静梅, 易健宏. 面向铜基复合材料文献的复杂实体关系抽取方法[J]. 计算机工程, 2025, 51(11): 100-111.
[6]	杨润, 陈艳平, 闫家鑫, 秦永彬. 基于关联邻接矩阵的关系抽取方法研究[J]. 计算机工程, 2025, 51(10): 121-129.
[7]	周雪阳, 傅启明, 陈建平, 陈延明, 陆悠, 王蕴哲. 基于证据和图推理的文档级关系抽取方法: 以医学关系为例[J]. 计算机工程, 2025, 51(1): 106-117.
[8]	曹渝昆, 程宇, 何祯奕, 徐康乐, 颜家洛, 李云峰. 文档上下文异构表示的句子级关系抽取方法[J]. 计算机工程, 2024, 50(5): 111-119.
[9]	李敬灿, 肖萃林, 覃晓婷, 谢夏. 基于大语言模型与语义增强的文本关系抽取算法[J]. 计算机工程, 2024, 50(4): 87-94.
[10]	吴海鹏, 钱育蓉, 冷洪勇. 基于双向注意力机制的多模态关系抽取[J]. 计算机工程, 2024, 50(4): 160-167.
[11]	陈志强, 仇瑜, 朱宇, 王晓英. 基于先验知识引导提示学习的自监督分类法补全[J]. 计算机工程, 2024, 50(12): 151-162.
[12]	廖涛, 张国畅, 张顺香. 基于双粒度图的文档级关系抽取[J]. 计算机工程, 2024, 50(10): 164-173.
[13]	马建红, 龚天, 姚爽. 基于证据句与图卷积网络的文档级关系抽取[J]. 计算机工程, 2023, 49(8): 104-110.
[14]	刘昊鑫, 董超, 勾智楠, 高凯. 融合混合表征的小样本关系抽取方法[J]. 计算机工程, 2023, 49(8): 63-68.
[15]	廖涛, 孙皓洁, 张顺香. 基于跨度和特征融合的实体关系联合抽取模型[J]. 计算机工程, 2023, 49(6): 107-114.

选择文件类型/文献管理软件名称

选择包含的内容

基于提示调优的少样本多模态关系抽取分层网络

Hierarchical Network for Few-shot Multimodal Relation Extraction Based on Prompt-tuning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 35

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于提示调优的少样本多模态关系抽取分层网络

Hierarchical Network for Few-shot Multimodal Relation Extraction Based on Prompt-tuning

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 35

相关文章 15

编辑推荐

Metrics

本文评价