基于预训练语言模型的关键词感知问题生成

doi:10.19678/j.issn.1000-3428.0060501

摘要/Abstract

摘要： 问题生成任务是指根据给定的文本段落和答案来自动生成对应的问题。针对现有问题生成方法存在的误差累积现象以及问题生成任务固有的“一对多”情况，提出一种带有关键词感知功能的问题生成方法。在预训练语言模型的基础上，实现关键词分类模型与问题生成模型的网络结构设计。输入文本段落中蕴含关键词，为使所生成的问题中包含同样的关键词以保证问题与段落的语义一致性，利用关键词分类模型提取出文本段落中的关键词，将关键词与非关键词的区分特征融入问题生成模型的输入中，该特征作为问题生成过程的全局信息，用以消除问题生成模型仅依赖局部最优解的弊端，减少误差累积与“一对多”情况的发生。在SQuAD数据集上的实验结果表明，该方法能够提升问题生成的质量，其BLEU-4指标值可达24，优于带有复制机制、带有语义监督的问题生成模型，目前已经借助百度百科数据平台实现了大规模工业应用。

关键词: 问题生成, 预训练语言模型, 关键词分类, 自注意力掩码, 嵌入向量

Abstract: The Question Generation(QG) task is to automatically generate the corresponding question based on a given text paragraph and answer.The existing QG methods often fail to deal with error accumulation and the one-answer-to-multiple-question problem in QG tasks.To address the problem, this paper proposes a keyword aware question generation method.We design the network structure for keyword classification and QG based on the pre-trained language model.To make the generated question include the same keywords as the input paragraph, which ensures the semantic consistency between the question and paragraph, we use the keyword classification model to extract the keywords in the paragraph, and integrate the feature that distinguish keywords from non-keywords into the input of the QG model.The feature acts as the global information of QG process to reduce dependency of the QG model on the local optimal solution only, and reduce the occurrence of error accumulation and one-answer-to-multiple-question problem.The experimental results on the SQuAD dataset show that this method can improve the quality of generated questions.Its BLEU-4 value reaches up to 24, higher than the QG models with replication mechanism or semantic supervision.This method has realized large-scale industrial application based on the Baidu Encyclopedia, a ten-million-scale data platform.

Key words: Question Generation(QG), pre-trained language model, keyword classification, self-attention mask, embedding vector

中图分类号:

TP18

于尊瑞, 毛震东, 王泉, 张勇东. 基于预训练语言模型的关键词感知问题生成[J]. 计算机工程, 2022, 48(2): 125-131.

YU Zunrui, MAO Zhendong, WANG Quan, ZHANG Yongdong. Keyword Aware Question Generation Based on Pre-Trained Language Model[J]. Computer Engineering, 2022, 48(2): 125-131.

http://www.ecice06.com/CN/Y2022/V48/I2/125

图/表 8

20220301123158

20220301123202

20220301123206

20220301123210

20220301123214

20220301123219

20220301123223

20220301123227

参考文献

[1] SHUM H Y, HE X, LI D.From Eliza to XiaoIce:challenges and opportunities with social chatbots[J].Frontiers of Information Technology & Electronic Engineering, 2018, 19(1):10-26.
[2] DENKOWSKI M, LAVIE A.Meteor universal:language specific translation evaluation for any target language[C]//Proceedings of the 9th Workshop on Statistical Machine Translation.[S.l.]:ACL Press, 2014:376-380.
[3] DANON G, LAST M.A syntactic approach to domain-specific automatic question generation[EB/OL].[2020-12-06].https://arxiv.org/abs/1712.09827.
[4] DU X, SHAO J, CARDIE C.Learning to ask:neural question generation for reading comprehension[EB/OL].[2020-12-06].https://arxiv.org/abs/1705.00106.
[5] HAN F X, NIU D, LAI K, et al.Inferring search queries from Web documents via a graph-augmented sequence to attention network[C]//Proceedings of the World Wide Web Conference.New York, USA:ACM Press, 2019:2792-2798.
[6] DU X, CARDIE C.Harvesting paragraph-level question-answer pairs from Wikipedia[EB/OL].[2020-12-06].https://arxiv.org/abs/1805.05942.
[7] TANG D, DUAN N, QIN T, et al.Question answering and question generation as dual tasks[EB/OL].[2020-12-06].https://arxiv.org/abs/1706.02027.
[8] TANG D, DUAN N, YAN Z, et al.Learning to collaborate for question answering and asking[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistics.[S.l.]:ACL Press, 2018:1564-1574.
[9] ALBERTI C, ANDOR D, PITLER E, et al.Synthetic QA corpora generation with roundtrip consistency[EB/OL].[2020-12-06].https://arxiv.org/abs/1906.05416.
[10] LINDBERG D, POPOWICH F, NESBIT J, et al.Generating natural language questions to support learning on-line[C]//Proceedings of the 14th European Workshop on Natural Language Generation.[S.l.]:ACL Press, 2013:105-114.
[11] MAZIDI K, NIELSEN R.Linguistic considerations in automatic question generation[C]//Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics.[S.l.]:ACL Press, 2014:321-326.
[12] HUSSEIN H, ELMOGY M, GUIRGUIS S.Automatic English question generation system based on template driven scheme[J].International Journal of Computer Science Issues, 2014, 11(6):45-46.
[13] LABUTOV I, BASU S, VANDERWENDE L.Deep questions without deep understanding[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.[S.l.]:ACL Press, 2015:889-898.
[14] ZHOU Q, YANG N, WEI F, et al.Neural question generation from text:a preliminary study[C]//Proceedings of National CCF Conference on Natural Language Processing and Chinese Computing.Berlin, Germany:Springer, 2017:662-671.
[15] KIM Y, LEE H, SHIN J, et al.Improving neural question generation using answer separation[C]//Proceedings of AAAI Conference on Artificial Intelligence.[S.l.]:AAAI Press, 2019:6602-6609.
[16] SCIALOM T, PIWOWARSKI B, STAIANO J.Self-attention architectures for answer-agnostic neural question generation[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.[S.l.]:ACL Press, 2019:6027-6032.
[17] ZHOU W, ZHANG M, WU Y.Question-type driven question generation[EB/OL].[2020-12-06].https://arxiv.org/abs/1909.00140.
[18] LIU B, WEI H, NIU D, et al.Asking questions the human way:scalable question-answer generation from text corpus[C]//Proceedings of 2020 Web Conference.New York, USA:ACM Press, 2020:2032-2043.
[19] LIU B, ZHAO M, NIU D, et al.Learning to generate questions by learning what not to generate[C]//Proceedings of the World Wide Web Conference.New York, USA:ACM Press, 2019:1106-1118.
[20] HU W, LIU B, MA J, et al.Aspect-based question generation[EB/OL].[2020-12-06].https://openreview.net/forum?id=rkRR1ynIf.
[21] GAO Y, BING L, CHEN W, et al.Difficulty controllable generation of reading comprehension questions[EB/OL].[2020-12-06].https://arxiv.org/abs/1807.03586.
[22] YAO K, ZHANG L, LUO T, et al.Teaching machines to ask questions[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence.Washington D.C., USA:IEEE Press, 2018:4546-4552.
[23] KUMAR V, RAMAKRISHNAN G, LI Y F.Putting the horse before the cart:a generator-evaluator framework for question generation from text[C]//Proceedings of the 23rd Conference on Computational Natural Language Learning.[S.l.]:CoNLL Press, 2019:812-821.
[24] ZHANG S, BANSAL M.Addressing semantic Drift in question generation for semi-supervised question answering[EB/OL].[2020-12-06].https://arxiv.org/abs/1909.06356.
[25] TANG D, DUAN N, QIN T, et al.Question answering and question generation as dual tasks[EB/OL].[2020-12-06].https://arxiv.org/pdf/1706.02027.pdf.
[26] WANG T, YUAN X, TRISCHLER A.A joint model for question answering and question generation[EB/OL].[2020-12-06].https://arxiv.org/abs/1706.01450.
[27] SUN Y, TANG D, DUAN N, et al.Joint learning of question answering and question generation[J].IEEE Transactions on Knowledge and Data Engineering, 2019, 32(5):971-982.
[28] DEVLIN J, CHANG M W, LEE K, et al.BERT:pre-training of deep bidirectional transformers for language understanding[EB/OL].[2020-12-06].https://arxiv.org/abs/1810.04805.
[29] SUN Y, WANG S, LI Y, et al.ERNIE:enhanced representation through knowledge integration[EB/OL].[2020-12-06].https://arxiv.org/abs/1904.09223.
[30] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all you need[EB/OL].[2020-12-06].https://papers.nips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-Paper.pdf.
[31] DONG L, YANG N, WANG W, et al.Unified language model pre-training for natural language understanding and generation[EB/OL].[2020-12-06].https://papers.nips.cc/paper/2019/file/c20bb2d9a50d5ac1f713f8b34d9aac5a-Paper. pdf.
[32] MA Y, YU D, WU T, et al.PaddlePaddle:an open-source deep learning platform from industrial practice[J].Frontiers of Data and Domputing, 2019, 1(1):105-115.
[33] RAJPURKAR P, ZHANG J, LOPYREV K, et al.SQuAD:100, 000+ questions for machine comprehension of text[EB/OL].[2020-12-06].https://arxiv.org/abs/1606.05250.
[34] ZHAO Y, NI X, DING Y, et al.Paragraph-level neural question generation with maxout pointer and gated self-attention networks[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing.[S.l.]:ACL Press, 2018:3901-3910.
[35] PAPINENI K, ROUKOS S, WARD T, et al.BLEU:a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics.[S.l.]:ACL Press, 2002:311-318.

选择文件类型/文献管理软件名称

选择包含的内容