基于知识架构的持续学习情感分类方法

doi:10.19678/j.issn.1000-3428.0063536

计算机工程 ›› 2023, Vol. 49 ›› Issue (2): 112-118. doi: 10.19678/j.issn.1000-3428.0063536

基于知识架构的持续学习情感分类方法

王松¹, 买日旦·吾守尔¹, 古兰拜尔·吐尔洪¹, 薛源^1,2

1. 新疆大学信息科学与工程学院, 乌鲁木齐 830046;
2. 清华大学电子工程系, 北京 100084

收稿日期:2021-12-15 修回日期:2022-03-03 发布日期:2022-06-30
作者简介:王松(1995-),男,硕士研究生,主研方向为持续学习、情感分类;买日旦·吾守尔(通信作者)、古兰拜尔·吐尔洪,副教授、博士;薛源,硕士研究生。
基金资助:
新疆维吾尔自治区自然科学基金（2021D01C118）；新疆维吾尔自治区高校科研计划项目（XJEDU2018Y005）。

Continual Learning Method for Sentiment Classification Based on Knowledge Architecture

WANG Song¹, Mairidan Wushouer¹, Gulanbaier Tuerhong¹, XUE Yuan^1,2

1. School of Information Science and Engineering, Xinjiang University, Urumqi 830046, China;
2. Department of Electronic Engineering, Tsinghua University, Beijing 100084, China

Received:2021-12-15 Revised:2022-03-03 Published:2022-06-30

摘要/Abstract

摘要： 当情感分类模型依次学习多个领域的情感分类任务时，从新任务中学到的参数会直接修改模型原有参数，由于缺少对原有参数的保护机制，降低了模型在旧任务上的分类准确率。为缓解灾难遗忘现象对模型性能的影响，并增加任务间的知识迁移，提出一种用于中文情感分类的基于知识架构的持续学习方法。在Transformer编码层中，采用任务自注意力机制为每个任务单独设置注意力变换矩阵，通过区分任务特有的注意力参数实现知识保留。在TextCNN的全连接层中，利用任务门控注意力（HAT）机制控制每个神经元的开闭，为每个任务训练特定的网络结构，仅激活对任务重要的神经元加强知识挖掘，提升分类效率与准确率。在JD21中文数据集上的实验结果表明，该方法的Last ACC和负类F1值相比于基于HAT的持续学习方法分别提升了0.37和0.09个百分点，具有更高的分类准确率，并且有效缓解了灾难遗忘现象。

关键词: 持续学习, 知识架构, 情感分类, 知识保留网络, 知识挖掘网络

Abstract: When a sentiment classification model learns sentiment classification tasks in multiple domains, the parameters learned from new tasks will modify the original parameters of the model.Because a protection mechanism for the original parameters does not exist, the classification accuracy of the model on old tasks is reduced.To alleviate the catastrophic forgetting of the sentiment classification model and increase knowledge transfer between tasks, this study proposes a Continual Learning(CL) method for sentiment classification based on a knowledge architecture.In the Transformer coding layer, the task self-attention mechanism is used to set the attention transformation matrix for each task separately, and knowledge is retained by distinguishing the task specific attention parameters. In the full connection layer of Convolutional Neural Networks for Sentence Classification(TextCNN), the Hard Attention on Task(HAT) mechanism is used to control the opening and closing of each neuron, train a specific network structure for each task, activate only the neurons important to the task to realize knowledge mining, and improve the classification efficiency and accuracy.Experimental results based on the JD21 Chinese dataset show that the Last Accuracy(Last ACC) and F1-scores of Negative classes(F1-NEG) of this method are 0.37 and 0.09 percentage points higher than those of the HAT-based CL method, respectively, which indicates the higher classification accuracy and effectiveness of the proposed method in mitigating catastrophic forgetting.

Key words: Continual Learning(CL), knowledge architecture, sentiment classification, Knowledge Retention Network(KRN), Knowledge Mining Network(KMN)

中图分类号:

TP183

王松, 买日旦·吾守尔, 古兰拜尔·吐尔洪, 薛源. 基于知识架构的持续学习情感分类方法[J]. 计算机工程, 2023, 49(2): 112-118.

WANG Song, Mairidan Wushouer, Gulanbaier Tuerhong, XUE Yuan. Continual Learning Method for Sentiment Classification Based on Knowledge Architecture[J]. Computer Engineering, 2023, 49(2): 112-118.

https://www.ecice06.com/CN/Y2023/V49/I2/112

图/表 7

20230216180707

20230216180711

20230216180714

20230216180717

20230216180720

20230216180724

20230216180728

参考文献

[1] THRUN S.Lifelong robot learning[J].Robotics and Autonomous Systems, 1995, 15(1/2):25-46.
[2] DE LANGE M, ALJUNDI R, MASANA M, et al.A continual learning survey:defying forgetting in classification tasks[J].IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(7):3366-3385.
[3] CHEN Z Y, MA N Z, LIU B.Lifelong learning for sentiment classification[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 2:Short Papers).Stroudsburg, USA:Association for Computational Linguistics, 2015:750-756.
[4] LOMONACO V.Continual learning with deep architectures[D].Bologna, Italy:University of Bologna, 2019.
[5] KE Z X, LIU B, WANG H, et al.Continual learning with knowledge transfer for sentiment classification[M].Berlin, Germany:Springer, 2021.
[6] CHEN Z Y, LIU B.Lifelong machine learning[J].Synthesis Lectures on Artificial Intelligence and Machine Learning, 2016, 10(3):101-145.
[7] BIESIALSKA M, BIESIALSKA K, COSTA-JUSSÀ M R.Continual lifelong learning in natural language processing:a survey[C]//Proceedings of the 28th International Conference on Computational Linguistics.Stroudsburg, USA:International Committee on Computational Linguistics, 2020:6523-6541.
[8] REBUFFI S A, KOLESNIKOV A, SPERL G, et al.iCaRL:incremental classifier and representation learning[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C., USA:IEEE Press, 2017:5533-5542.
[9] LOPEZ-PAZ D, RANZATO M.Gradient episodic memory for continual learning[C]//Proceedings of NIPS'17.Long Beach, USA:Curran Associates Inc., 2017:6470-6479.
[10] D'AUTUME C D M, RUDER S, KONG L, et al.Episodic memory in lifelong language learning[EB/OL].[2021-11-05].https://arxiv.org/abs/1906.01076v1.
[11] SHIN H, LEE J K, KIM J, et al.Continual learning with deep generative replay[EB/OL].[2021-11-05].https://arxiv.org/abs/1705.08690.
[12] SUN F K, HO C H, LEE H Y.LAMOL:language modeling for lifelong language learning[EB/OL].[2021-11-05].https://arxiv.org/abs/1909.03329.
[13] KIRKPATRICK J, PASCANU R, RABINOWITZ N, et al.Overcoming catastrophic forgetting in neural networks[J].Proceedings of the National Academy of Sciences, 2017, 114(13):3521-3526.
[14] LEE S W, KIM J H, JUN J, et al.Overcoming catastrophic forgetting by incremental moment matching[EB/OL].[2021-11-05].https://arxiv.org/abs/1703.08475.
[15] DHAR P, SINGH R V, PENG K C, et al.Learning without memorizing[C]//Proceedings of IEEE/CVF Conference on Computer Vision and Pattern Recognition(CVPR).Washington D.C., USA:IEEE Press, 2019:5133-5141.
[16] JUNG H, JU J, JUNG M, et al.Less-forgetting learning in deep neural networks[EB/OL].[2021-11-05].https://arxiv.org/abs/1607.00122.
[17] FERNANDO C, BANARSE D, BLUNDELL C, et al.PathNet:evolution channels gradient descent in super neural networks[EB/OL].[2021-11-05].https://arxiv.org/abs/1701.08734.
[18] SERRÀ J, SURÍS D, MIRON M, et al.Overcoming catastrophic forgetting with hard attention to the task[EB/OL].[2021-11-05].https://arxiv.org/abs/1801.01423.
[19] RUSU A A, RABINOWITZ N C, DESJARDINS G, et al.Progressive neural networks[EB/OL].[2021-11-05].https://arxiv.org/abs/1606.04671.
[20] WANG H, LIU B, WANG S, et al.Forward and backward knowledge transfer for sentiment classification[EB/OL].[2021-11-05].https://arxiv.org/abs/1906.03506.
[21] 买日旦∙吾守尔, 古兰拜尔∙吐尔洪, 王松.基于领域注意力机制的终身朴素贝叶斯文本分类方法:CN202110427951.7[P].2021-06-29. Mairidan Wushouer, Gulanbaier Tuerhong, WANG S.Sentiment classification based on lifelong naive Bayesian:CN202110427951.7[P].2021-06-29.(in Chinese)
[22] LÜ G Y, WANG S, LIU B, et al.Sentiment classification by leveraging the shared knowledge from a sequence of domains[M].Berlin, Germany:Springer, 2019.
[23] WANG H, WANG S, MAZUMDER S, et al.Bayes-enhanced lifelong attention networks for sentiment classification[C]//Proceedings of the 28th International Conference on Computational Linguistics.Stroudsburg, USA:International Committee on Computational Linguistics, 2020:580-591.
[24] VASWANI A, SHAZEER N, PARMAR N, et al.Attention is all You need[EB/OL].[2021-11-05].https://arxiv.org/abs/1706.03762.
[25] KIM Y.Convolutional neural networks for sentence classification[EB/OL].[2021-11-05].https://arxiv.org/abs/1408.5882.
[26] TOLSTIKHIN I, HOULSBY N, KOLESNIKOV A, et al.MLP-Mixer:an all-MLP architecture for vision[EB/OL].[2021-11-05].https://arxiv.org/abs/2105.01601.
[27] AHN H, CHA S, LEE D, et al.Uncertainty-based continual learning with adaptive regularization[EB/OL].[2021-11-05].https://arxiv.org/abs/1905.11614.
[28] YANG B, LI J, WONG D, et al.Context-aware self-attention networks[EB/OL].[2021-11-05].https://arxiv.org/abs/1902.05766v1.

选择文件类型/文献管理软件名称

选择包含的内容

基于知识架构的持续学习情感分类方法

Continual Learning Method for Sentiment Classification Based on Knowledge Architecture

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	宋华伟, 李升起, 万方杰, 卫玉萍. 非独立同分布场景下的联邦学习优化方法[J]. 计算机工程, 2024, 50(3): 166-172.
[2]	崔蒙蒙, 刘井平, 阮彤, 宋雨秋, 杜渂. 基于双重多视角表示的目标级隐性情感分类[J]. 计算机工程, 2024, 50(1): 79-90.
[3]	蒋慧凯, 李晓戈, 安晓春, 王甜甜, 阮桁. 面向属性情感分类的无标签数据增强位置感知网络[J]. 计算机工程, 2023, 49(11): 106-114.
[4]	吴迪, 王梓宇, 赵伟超. ELMo-CNN-BiGRU双通道文本情感分类模型[J]. 计算机工程, 2022, 48(8): 105-112.
[5]	王义, 沈洋, 戴月明. 基于细粒度多通道卷积神经网络的文本情感分析[J]. 计算机工程, 2020, 46(5): 102-108.
[6]	胡均毅, 李金龙. 基于情感评分的分层文本表示情感分类方法[J]. 计算机工程, 2020, 46(3): 46-52,59.
[7]	陈景霞,郑茹,张鹏伟,贾小云. 基于数据空间自适应与共空间模式的脑电情感分类[J]. 计算机工程, 2019, 45(4): 296-301.
[8]	周锦峰,叶施仁,王晖. 基于深度卷积神经网络模型的文本情感分类[J]. 计算机工程, 2019, 45(3): 300-308.
[9]	吕超,杨超,李仁发. 基于CNN与词语邻近特征的情感分类模型[J]. 计算机工程, 2018, 44(5): 182-187.
[10]	陈炳丰,郝志峰,蔡瑞初,温雯,梁礼欣. 基于AWCRF模型的微博情感倾向分类方法[J]. 计算机工程, 2017, 43(7): 187-192.
[11]	李平,戴月明,王艳. 基于混合卡方统计量与逻辑回归的文本情感分析[J]. 计算机工程, 2017, 43(12): 192-196,202.
[12]	郑诚,沈磊,代宁. 基于类序列规则的中文微博情感分类[J]. 计算机工程, 2016, 42(2): 184-189,194.
[13]	王鹏程,肖正,刘辉. 融合观点倾向的话题热度趋势建模研究[J]. 计算机工程, 2015, 41(7): 66-70.
[14]	周剑峰，阳爱民，周咏梅，王璇璇. 基于二元搭配词的微博情感特征选择[J]. 计算机工程, 2014, 40(6): 162-165.
[15]	热依莱木·帕尔哈提,孟祥涛,艾斯卡尔·艾木都拉. 基于区分性关键词模型的维吾尔文本情感分类[J]. 计算机工程, 2014, 40(10): 132-136,142.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于知识架构的持续学习情感分类方法

Continual Learning Method for Sentiment Classification Based on Knowledge Architecture

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献

相关文章 15

编辑推荐

Metrics

本文评价