Abstract:

Large Language Models (LLMs) have made significant progress in dialogue, reasoning, and knowledge retention. However, they still face challenges in terms of factual accuracy, knowledge updates, and a lack of high-quality domain datasets for handling knowledge-intensive tasks in the electricity sector. This study aims to address these challenges by introducing an improved Retrieval-Augmented Generation (RAG) strategy. This strategy combines hybrid retrieval with a fine-tuned generative model for efficient knowledge capturing and updating. The Metadata-driven RAG framework (Meta-RAG) is proposed for knowledge Question Answering (QA) tasks in the electricity domain. This includes data preparation, model fine-tuning, and reasoning retrieval stages. The data-preparation stage involves document conversion, metadata extraction and enhancement, and document parsing. These processes ensure efficient indexing and structured processing of power regulation documents. The Electricity Question Answering (EleQA) dataset, consisting of 19 560 QA pairs, is constructed specifically for this sector. The model fine-tuning stage uses multi-question generation, chain-of-thought prompting, and supervised instruction fine-tuning to optimize the reasoning abilities in specific tasks. The retrieval reasoning stage employs mixed encoding and re-ranking strategies, combining retrieval and generation modules to improve answer accuracy and relevance. Experiments validate the effectiveness of Meta-RAG. Compared to baseline models such as Self-RAG, Corrective-RAG, Adaptive-RAG, and RA-ISF, Meta-RAG shows higher answer accuracy and retrieval hit rates. Meta-RAG with the Qwen1.5-14B-Chat model achieves an overall accuracy of 0.804 3, surpassing the other methods. Ablation and document recall experiments indicate that document retrieval significantly impacts the framework performance, with a 0.292 8 drop in accuracy when the retrieval capability is lost.

Key words: EleQA dataset, meta-information extraction, knowledge Question Answering (QA), power industry, Retrieval-Augmented Generation (RAG), model fine-tuning, document conversion

摘要：

大语言模型(LLM)在对话、推理和知识保留能力方面展现了显著优势, 但在处理电力领域知识密集型任务时仍面临事实准确性不足、知识更新难以及高质量领域数据集匮乏的问题。针对这些挑战, 引入一种改进的检索增强生成(RAG)策略, 该策略融合了混合检索策略和经过微调的生成模型, 提供了更高效的知识捕获和更新能力。基于对现有方法的深入分析, 针对电力领域的知识问答(QA)任务, 提出了元数据驱动的RAG框架Meta-RAG, 该框架包含数据准备、模型微调和检索推理3个阶段。数据准备阶段包括文档转换、元信息抽取与增强及文档解析模块, 在此阶段, 借助元信息的提取与增强确保了电力规范文档的高效索引和结构化处理, 并且构建了电力领域的EleQA(Electricity Question Answering)数据集, 这是一个包含19 560个问答对的电力规范问答数据集。在模型微调阶段, 通过多问题生成、思维链提示生成和监督指令微调数据集构建模块, 优化了模型在特定电力问答任务上的推理能力。在检索推理阶段则采用混合编码和重排序策略, 结合检索和生成模块, 进一步提高了答案的准确性和合理性。通过一系列实验, Meta-RAG的有效性得到验证。与Self-RAG、Corrective-RAG、Adaptive-RAG、RA-ISF等基线模型相比, Meta-RAG具有更高的回答准确率和检索命中率, 其中, 基于Qwen1.5-14B-Chat模型的Meta-RAG达到了整体准确率0.804 3, 高于其他方法。消融实验和文档召回实验结果表明文档检索对框架性能影响最大, 失去检索能力整体准确率下降了0.292 8。

关键词: EleQA数据集, 元信息抽取, 知识问答, 电力领域, 检索增强生成, 模型微调, 文档转换