基于深度学习的文本分类技术研究进展

doi:10.19678/j.issn.1000-3428.0059099

摘要/Abstract

摘要： 随着深度学习技术的快速发展，许多研究者尝试利用深度学习来解决文本分类问题，特别是在卷积神经网络和循环神经网络方面，出现了许多新颖且有效的分类方法。对基于深度神经网络的文本分类问题进行分析，介绍卷积神经网络、循环神经网络、注意力机制等方法在文本分类中的应用和发展，分析多种典型分类方法的特点和性能，从准确率和运行时间方面对基础网络结构进行比较，表明深度神经网络较传统机器学习方法在用于文本分类时更具优势，其中卷积神经网络具有优秀的分类性能和泛化能力。在此基础上，指出当前深度文本分类模型存在的不足，并对未来的研究方向进行展望。

关键词: 深度学习, 文本分类, 卷积神经网络, 循环神经网络, 注意力机制

Abstract: With the rapid development of deep learning technology,many researchers have tried to utilize deep learning to solve the text classification problems.Especially in terms of the Convolutional Neural Network(CNN) and Recurrent Neural Network(RNN),many novel and effective classification methods have been proposed.This paper analyzes the problem of text classification based on Deep Neural Network(DNN) and introduces the application and development of the CNN,RNN,attention mechanism and other methods in text classification.Then it analyzes the characteristics and performance of multiple text classification methods based on deep learning,and compares the basic network structures in terms of accuracy and elapsed time,and shows that DNN methods outperform the traditional machine learning methods,and among them,CNN provides better classification performance and generalization ability.On this basis,this paper points out the deficiencies of the existing deep models for text classification,and prospects the direction of future research.

Key words: deep learning, text classification, Convolutional Neural Network(CNN), Recurrent Neural Network(RNN), attention mechanism

中图分类号:

TP18

何力, 郑灶贤, 项凤涛, 吴建宅, 谭林. 基于深度学习的文本分类技术研究进展[J]. 计算机工程, 2021, 47(2): 1-11.

HE Li, ZHENG Zaoxian, XIANG Fengtao, WU Jianzhai, TAN Lin. Research Progress of Text Classification Technology Based on Deep Learning[J]. Computer Engineering, 2021, 47(2): 1-11.

https://www.ecice06.com/CN/Y2021/V47/I2/1

图/表 5

20210225090614

20210225090618

20210225090621

20210225090625

20210225090628

参考文献 56

[1]	LI H,YAMANISHI K.Text classification using ESC-based stochastic decision lists[C]//Proceedings of the 8th International Conference on Information and Knowledge Management.New York,USA:ACM Press,1999:122-130.
[2]	KIM S B,HAN K S,RIM H C,et al.Some effective techniques for naive Bayes text classification[J].IEEE Transactions on Knowledge and Data Engineering,2006,18(11):1457-1466.
[3]	JOACHIMS T.Text categorization with support vector machines:learning with many relevant features[M].New York,USA:Springer US,1998.
[4]	KAZAMA J,TSUJII J.Maximum entropy models with inequality constraints:a case study on text categorization[J].Machine Learning,2005,60(1/2/3):159-194.
[5]	SOUCY P,MINEAU G W.A simple KNN algorithm for text categorization[C]//Proceedings of 2001 IEEE Inter-national Conference on Data Mining.Washington D.C.,USA:IEEE Press,2001:647-648.
[6]	PENNINGTON J,SOCHER R,MANNING C.GloVe:global vectors for word representation[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2014:1532-1543.
[7]	MIKOLOV T,CHEN K,CORRADO G,et al.Efficient estimation of word representations in vector space[EB/OL].(2013-01-16)[2020-08-10].https://arxiv.org/pdf/1301.3781.pdf.
[8]	BENGIO Y,DUCHARME R,VINCENT P,et al.A neural probabilistic language model[J].Journal of Machine Learning Research,2003,3:1137-1155.
[9]	MIKOLOV T,SUTSKEVER I,CHEN K,et al.Distributed representations of words and phrases and their composition-ality[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2013:3111-3119.
[10]	LAI Siwei.Research on word and document semantic vector representation methods based on neural network[D].Beijing:University of Chinese Academy of Sciences,2016.(in Chinese)来斯惟.基于神经网络的词和文档语义向量表示方法研究[D].北京:中国科学院大学,2016.
[11]	KALCHBRENNER N,GREFENSTETTE E,BLUNSOM P.A convolutional neural network for modelling sentences[EB/OL].(2014-04-08)[2020-08-10].https://arxiv.org/pdf/1404.2188.pdf.
[12]	KIM Y.Convolutional neural networks for sentence classification[C]//Proceedings of 2014 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2014:1746-1751.
[13]	ZHANG X,ZHAO J B,LECUN Y.Character-level convolutional networks for text classification[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:649-657.
[14]	JOULIN A,GRAVE E,BOJANOWSKI P,et al.Bag of tricks for efficient text classification[C]//Proceedings of the 15th Conference of the European Chapter of the Association for Computational Linguistics(Volume 2:Short Papers).San Diego,USA:ACL,2017:427-431.
[15]	JOHNSON R,ZHANG T.Deep pyramid convolutional neural networks for text categorization[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Volume 1:Long Papers).San Diego,USA:ACL,2017:562-570.
[16]	HONG S,OH J,HAN B,et al.Learning transferrable knowledge for semantic segmentation with deep convolutional neural network[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:3204-3212.
[17]	CONNEAU A,SCHWENK H,BARRAULT L,et al.Very deep convolutional networks for natural language processing[EB/OL].(2016-06-06)[2020-08-10].https://arxiv.org/pdf/1606.01781v1.pdf.
[18]	HE Kaiming,ZHANG Xiangyu,REN Shaoqing,et al.Deep residual learning for image recognition[C]//Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2016:770-778.
[19]	LE H,CERISARA C,DENIS A.Do convolutional networks need to be deep for text classification?[C]//Proceedings of AAAI Workshop on Affective Content Analysis.[S.l.]:AAAI,2018:29-36.
[20]	HUANG G,LIU Z,MAATEN V,et al.Densely connected convolutional networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:770-778.
[21]	ZHANG Y,WALLACE B.A sensitivity analysis of (and practitioners' guide to) convolutional neural networks for sentence classification[EB/OL].(2015-10-13)[2020-08-10].https://arxiv.org/pdf/1510.03820.pdf.
[22]	HOCHREITER S,SCHMIDHUBER J.Long short-term memory[J].Neural Computation,1997,9(8):1735-1780.
[23]	GRAVES A.Supervised sequence labelling with recurrent neural networks[M].Berlin,Germany:Springer,2012.
[24]	CHO K,MERRIENBOER B,GULCEHRE C,et al.Learning phrase representations using RNN encoder-decoder for statistical machine translation[EB/OL].(2014-06-03)[2020-08-10].https://arxiv.org/pdf/1406.1078.pdf.
[25]	YU K Y,LIU Y,SCHWING A,et al.Fast and accurate text classification:skimming,rereading and early stopping[C]//Proceedings of the 6th International Conference on Learning Representations.Vancouver,Canada:[s.n.],2018:1-5.
[26]	LIU Pengfei,QIU Xipeng,HUANG Xuanjing.Recurrent neural network for text classification with multi-task learning[C]//Proceedings of International Joint Conference on Artificial Intelligence.[S.l.]:AAAI,2016:2873-2879.
[27]	DAI A,LE Q V.Semi-supervised sequence learning[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:3079-3087.
[28]	LAI Siwei,XU Liheng,LIU Kang,et al.Recurrent convolutional neural networks for text classification[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence.[S.l.]:AAAI,2015:2267-2273.
[29]	XIAO Y J,CHO K.Efficient character-level document classification by combining convolution and recurrent layers[EB/OL].(2016-02-01)[2020-08-10].https://arxiv.org/pdf/1602.00367.pdf.
[30]	HASSAN A,MAHMOOD A.Convolutional recurrent deep learning model for sentence classification[J].IEEE Access,2018,6:13949-13957.
[31]	CHEN Guibin,YE Deheng,XING Zhenchang,et al.Ensemble application of convolutional and recurrent neural networks for multi-label text categorization[C]//Proceedings of 2017 International Joint Conference on Neural Networks.Washington D.C.,USA:IEEE Press,2017:2377-2383.
[32]	TOM Y,DEVAMANYU H,SOUJANYA P,et al.Recent trends in deep learning based natural language pro-cessing[J].IEEE Computational Intelligence Magazine,2018,13(3):55-75.
[33]	BAHDANAU D,CHO K,BENGIO Y.Neural machine translation by jointly learning to align and translate[C]//Proceedings of the 3rd International Conference on Learning Representations.Banff,Canada:[s.n.],2014:1-5.
[34]	XU K,BA J,KIROS R,et al.Show,attend and tell:neural image caption generation with visual attention[C]//Proceedings of the 32nd International Conference on Machine Learning.[S.l.]:JMLR,2015:2048-2057.
[35]	HERMANN K,KOISK T,GREFENSTETTE E,et al.Teaching machines to read and comprehend[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2015:1693-1701.
[36]	YANG Z C,YANG D Y,DYER C,et al.Hierarchical attention networks for document classification[C]//Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics Human Language Technologies:Human Language Tech-nologies.San Diego,USA:ACL,2016:1480-1489.
[37]	FAN Feifan,FENG Yansong,ZHAO Dongyan.Multi-grained attention network for aspect-level sentiment classification[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2018:3433-3442.
[38]	WANG Yequan,HUANG Minlie,ZHU Xiaoyan,et al.Attention-based LSTM for aspect-level sentiment classifi-cation[C]//Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2016:606-615.
[39]	MA Y K,PENG H Y,CAMBRIA E.Targeted aspect-based sentiment analysis via embedding commonsense knowledge into an attentive LSTM[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence.[S.l.]:AAAI,2018:5876-5883.
[40]	LI X,BING L D,LAM W,et al.Transformation networks for target-oriented sentiment classification[C]//Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics.San Diego,USA:ACL,2018:946-956.
[41]	TANG Duyu,QIN Bing,LIU Ting.Aspect level sentiment classification with deep memory network[C]//Proceedings of 2016 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2016:214-224.
[42]	YAO Liang,MAO Chengsheng,LUO Yuan.Graph convolutional networks for text classification[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence.[S.l.]:AAAI,2019:7370-7377.
[43]	QIAO Chao,HUANG Bo,NIU Guocheng,et al.A new method of region embedding for text classification[C]//Proceedings of the 6th International Conference on Learning Representations.Vancouver,Canada,2018:6-10.
[44]	XIANG Liuyu,JIN Xiaoming,YI Lan,et al.Adaptive region embedding for text classification[C]//Proceedings of the 33rd AAAI Conference on Artificial Intelligence.[S.l.]:AAAI,2019:7314-7321.
[45]	BAO Yujia,WU Menghua,CHANG Shiyu,et al.Few-shot text classification with distributional signatures[C]//Proceedings of the 8th International Conference on Learning Representation.Addis Ababa,Ethiopia:[s.n.],2020:1-5.
[46]	JAKE S,KEVIN S,ZEMEL R.Prototypical networks for few-shot learning[C]//Proceedings of the 30th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2017:4077-4087.
[47]	ZHANG Tianyang,HUANG Minlie,ZHAO Li.Learning structured representation for text classification via reinforcement learning[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence.[S.l.]:AAAI,2018:6053-6060.
[48]	KOWSARI K,HEIDARYSAFA M,BROWN D E,et al.RMDL:random multimodel deep learning for classification[C]//Proceedings of the 2nd International Conference on Information System and Data Mining.New York,USA:ACM Press,2018:19-28.
[49]	TANG Duyu,QIN Bing,LIU Ting.Document modeling with gated recurrent neural network for sentiment classification[C]//Proceedings of 2015 Conference on Empirical Methods in Natural Language Processing.San Diego,USA:ACL,2015:1422-1432.
[50]	CHEN Ke,LIANG Bin,KE Wende,et al.Chinese micro-blog sentiment analysis based on multi-channels convolutional neural networks[J].Journal of Computer Research and Development,2018,55(5):945-957.(in Chinese)陈珂,梁斌,柯文德,等.基于多通道卷积神经网络的中文微博情感分析[J].计算机研究与发展,2018,55(5):945-957.
[51]	GOODFELLOW I,POUGET-ABADIE J,MIRZA M,et al.Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems.Cambridge,USA:MIT Press,2014:2672-2680.
[52]	DAI B,FIDLER S,URTASUN R,et al.Towards diverse and natural image descriptions via a conditional GAN[C]//Proceedings of 2017 IEEE International Conference on Computer Vision.Washington D.C.,USA:IEEE Press,2017:2989-2998.
[53]	YANG Zhen,CHEN Wei,WANG Feng,et al.Improving neural machine translation with conditional sequence generative adversarial nets[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.San Diego,USA:ACL,2018:1346-1355.
[54]	ZHANG Han,XU Tao,LI Hongsheng,et al.StackGAN:text to photo-realistic image synthesis with stacked generative adversarial networks[C]//Proceedings of 2017 IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2017:5908-5916.
[55]	XU Tao,ZHANG Pengchuan,HUANG Qiuyuan,et al.AttnGAN:fine-grained text to image generation with attentional generative adversarial networks[C]//Proceedings of 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2018:1316-1324.
[56]	QIAO Tingting,ZHANG Jing,XU Duanqing,et al.MirrorGAN:learning text-to-image generation by redescrip-tion[C]//Proceedings of 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2019:1505-1514.

选择文件类型/文献管理软件名称

选择包含的内容