Meta-RAG:基于元数据驱动的电力领域检索增强生成框架

doi:10.19678/j.issn.1000-3428.0070415

摘要/Abstract

摘要： 大语言模型(LLM)在对话、推理和知识保留能力方面展现了显著优势,但在处理电力领域知识密集型任务时仍面临事实准确性不足、知识更新难以及高质量领域数据集匮乏的问题。针对这些挑战,引入一种改进的检索增强生成(RAG)策略,该策略融合了混合检索策略和经过微调的生成模型,提供了更高效的知识捕获和更新能力。基于对现有方法的深入分析,针对电力领域的知识问答(QA)任务,提出了元数据驱动的RAG框架Meta-RAG,该框架包含数据准备、模型微调和检索推理3个阶段。数据准备阶段包括文档转换、元信息抽取与增强及文档解析模块,在此阶段,借助元信息的提取与增强确保了电力规范文档的高效索引和结构化处理,并且构建了电力领域的EleQA(Electricity Question Answering)数据集,这是一个包含19 560个问答对的电力规范问答数据集。在模型微调阶段,通过多问题生成、思维链提示生成和监督指令微调数据集构建模块,优化了模型在特定电力问答任务上的推理能力。在检索推理阶段则采用混合编码和重排序策略,结合检索和生成模块,进一步提高了答案的准确性和合理性。通过一系列实验,Meta-RAG的有效性得到验证。与Self-RAG、Corrective-RAG、Adaptive-RAG、RA-ISF等基线模型相比,Meta-RAG具有更高的回答准确率和检索命中率,其中,基于Qwen1.5-14B-Chat模型的Meta-RAG达到了整体准确率0.804 3,高于其他方法。消融实验和文档召回实验结果表明文档检索对框架性能影响最大,失去检索能力整体准确率下降了0.292 8。

关键词: EleQA数据集, 元信息抽取, 知识问答, 电力领域, 检索增强生成, 模型微调, 文档转换

Abstract: Large Language Models (LLMs) have made significant progress in dialogue, reasoning, and knowledge retention. However, they still face challenges in terms of factual accuracy, knowledge updates, and a lack of high-quality domain datasets for handling knowledge-intensive tasks in the electricity sector. This study aims to address these challenges by introducing an improved Retrieval-Augmented Generation (RAG) strategy. This strategy combines hybrid retrieval with a fine-tuned generative model for efficient knowledge capturing and updating. The Metadata-driven RAG framework (Meta-RAG) is proposed for knowledge Question Answering (QA) tasks in the electricity domain. This includes data preparation, model fine-tuning, and reasoning retrieval stages. The data-preparation stage involves document conversion, metadata extraction and enhancement, and document parsing. These processes ensure efficient indexing and structured processing of power regulation documents. The Electricity Question Answering (EleQA) dataset, consisting of 19 560 QA pairs, is constructed specifically for this sector. The model fine-tuning stage uses multi-question generation, chain-of-thought prompting, and supervised instruction fine-tuning to optimize the reasoning abilities in specific tasks. The retrieval reasoning stage employs mixed encoding and re-ranking strategies, combining retrieval and generation modules to improve answer accuracy and relevance. Experiments validate the effectiveness of Meta-RAG. Compared to baseline models such as Self-RAG, Corrective-RAG, Adaptive-RAG, and RA-ISF, Meta-RAG shows higher answer accuracy and retrieval hit rates. Meta-RAG with the Qwen1.5-14B-Chat model achieves an overall accuracy of 0.804 3, surpassing the other methods. Ablation and document recall experiments indicate that document retrieval significantly impacts the framework performance, with a 0.292 8 drop in accuracy when the retrieval capability is lost.

Key words: EleQA dataset, meta-information extraction, knowledge Question Answering (QA), power industry, Retrieval-Augmented Generation (RAG), model fine-tuning, document conversion

中图分类号:

TP18

王合庆, 魏杰, 景红雨, 宋晖, 徐波. Meta-RAG:基于元数据驱动的电力领域检索增强生成框架[J]. 计算机工程, 2026, 52(2): 383-392.

WANG Heqing, WEI Jie, JING Hongyu, SONG Hui, XU Bo. Meta-RAG: A Metadata-Driven Retrieval-Augmented Generation Framework for the Power Industry[J]. Computer Engineering, 2026, 52(2): 383-392.

https://www.ecice06.com/CN/Y2026/V52/I2/383

参考文献

[1] LEWIS P, PEREZ E, PIKTUS A, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks[C]//Proceedings of Conference on Neural Information Processing Systems. Berlin, Germany: Springer, 2020: 1-10.
[2] KARPUKHIN V, O AGˇG UZ B, MIN S, et al. Dense passage retrieval for open-domain question answering[EB/OL].[2024-07-27]. https://arxiv.org/abs/2004.04906.
[3] IZACARD G, LEWIS P, LOMELI M, et al. Atlas: few-shot learning with retrieval augmented language models[J]. Journal of Machine Learning Research, 2023, 24(251): 1-43.
[4] YAO J Y, NING K P, LIU Z H, et al. LLM lies: hallucinations are not bugs, but features as adversarial examples[EB/OL].[2024-07-27]. https://arxiv.org/abs/2310.01469.
[5] BANG Y, CAHYAWIJAYA S, LEE N, et al. A multitask, multilingual, multimodal evaluation of ChatGPT on reasoning, hallucination, and interactivity[EB/OL].[2024-07-27]. https://arxiv.org/abs/2302.04023.
[6] ASAI A, WU Z, WANG Y, et al. Self-RAG: learning to retrieve, generate, and critique through self-reflection[C]//Proceedings of the 12th International Conference on Learning Representations. Berlin, Germany: Springer, 2024: 1-10.
[7] YAN S Q, GU J C, ZHU Y, et al. Corrective retrieval augmented generation[EB/OL].[2024-07-27]. https://arxiv.org/abs/2401.15884.
[8] JEONG S, BAEK J, CHO S, et al. Adaptive-RAG: learning to adapt retrieval-augmented large language models through question complexity[C]//Proceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers). Stroudsburg, USA: ACL, 2024: 7036-7050.
[9] LIU Y M, PENG X Y, ZHANG X H, et al. RA-ISF: learning to answer and understand from retrieval augmentation via iterative self-feedback[EB/OL].[2024-07-27]. https://arxiv.org/abs/2403.06840.
[10] 张峰, 杨晓艺, 刘奕湘. 电力智能问答平台架构的研究与设计[J]. 能源与环保, 2017, 39(7): 193-195. ZHANG F, YANG X Y, LIU Y X. Research and design of power intelligent question-answering platform architecture[J]. Energy and Environmental Protection, 2017, 39(7): 193-195. (in Chinese)
[11] 周帆, 叶健辉, 肖林朋, 等. 基于知识图谱的电网模型本体智能问答系统研究[J]. 中国科技信息, 2019, 31(16): 85-86. ZHOU F, YE J H, XIAO L P, et al. Research on intelligent question-answering system of power grid model ontology based on knowledge graph[J]. China Science and Technology Information, 2019, 31(16): 85-86. (in Chinese)
[12] 覃祥坤. 一种电力图谱问答系统设计与实现[D]. 北京:中国科学院大学人工智能学院, 2020. QIN X K. Design and implementation of a power graph question answering system[D]. Beijing: School of Artificial Intelligence, University of Chinese Academy of Sciences, 2020. (in Chinese)
[13] ZHANG Q, JIA Q Y, WANG Y H. Question answering based assisted decision for electric power fault diagnosis[C]//Proceedings of the IEEE 5th International Conference on Cloud Computing and Big Data Analytics. Chengdu, China: IEEE Press, 2020: 194-198.
[14] MENG F Q, WANG W H, WANG J D. Research on short text similarity calculation method for power intelligent question answering[C]//Proceedings of the 13th International Conference on Computational Intelligence and Communication Networks. Lima, Peru: IEEE Press, 2021: 91-95.
[15] LI W Q, QI X M, ZHAO Q, et al. Knowledge graph-based credibility evaluation method for electric grid large language model knowledge question-answering[C]//Proceedings of the 7th International Conference on Electronic Information Technology and Computer Engineering. New York, USA: ACM, 2023: 754-759.
[16] XIN R, ZHANG P F, CHEN X, et al. Knowledge graph question-answering based on link reasoning for electrical equipment[C]//Proceedings of the 2024 International Conference on Power Electronics and Artificial Intelligence. New York, USA: ACM, 2024: 594-600.
[17] ZHAO J X, MA Z C, ZHAO H, et al. Self-Consistency, Extract and Rectify: knowledge graph enhance large language model for electric power question answering[C]//Advanced Intelligent Computing Technology and Applications. Singapore: Springer Nature Singapore, 2024: 493-504.
[18] HUANG C H, LI S Y, LIU R H, et al. Large foundation models for power systems[EB/OL].[2024-07-27]. https://arxiv.org/abs/2312.07044.
[19] CHENG Y H, ZHAO H, ZHOU X Y, et al. A large language model for advanced power dispatch[EB/OL].[2024-07-27]. https://arxiv.org/abs/2408.03847.
[20] MAJUMDER S, DONG L, DOUDI F, et al. Exploring the capabilities and limitations of large language models in the electric energy sector[J]. Joule, 2024, 8(6): 1544-1549.
[21] RUAN J Q, LIANG G Q, ZHAO H, et al. Applying large language models to power systems: potential security threats[J]. IEEE Transactions on Smart Grid, 2024, 15(3): 3333-3336.
[22] HU E J, SHEN Y, WALLIS P, et al. LoRA: low-rank adaptation of large language models[EB/OL].[2024-07-27]. https://arxiv.org/abs/2106.09685.
[23] ZHANG T J, PATIL S G, JAIN N, et al. RAFT: adapting language model to domain specific RAG[EB/OL].[2024-07-27]. https://arxiv.org/abs/2403.10131.
[24] CUCONASU F, TRAPPOLINI G, SICILIANO F, et al. The power of noise: redefining retrieval for RAG systems[EB/OL].[2024-07-27]. https://arxiv.org/abs/2401.14887.
[25] DMONTE A, ORUCHE R, ZAMPIERI M, et al. Claim verification in the age of large language models: a survey[EB/OL].[2024-07-27]. https://arxiv.org/abs/2408.14317.

选择文件类型/文献管理软件名称

选择包含的内容