Visual Sentiment Analysis Based on Multi-level Features Fusion of Dual Attention

doi:10.19678/j.issn.1000-3428.0058303

Abstract

Abstract: In order to obtain more discriminative image features and improve the accuracy of sentiment classification, a visual sentiment analysis model based on multi-level feature fusion of dual attention is proposed.Firstly, the model extracts multi-channel and multi-level features of images with Convolution Neural Network(CNN).Secondly, the extracted low-level features are given spatial attention weight with spatial attention mechanism, and the extracted high-level features are given channel attention weight with channel attention mechanism to saparately enhance the low and high level features.And finally, the enhaned high-level features and low-level features are fused to form discriminant features for training sentiment classifiers.Experiments on three real datasets TwitterⅠ, TwitterⅡ and EmotionROI showed that the classification accuracy of the method reached 79.83%, 78.25% and 49.34%, which improved the effect of social media visual sentiment analysis.

Key words: social media, visual sentiment analysis, Convolutional Neural Network(CNN), attention, features fusion

摘要： 为获得更具判别性的视觉特征并提升情感分类效果，构建融合双注意力多层特征的视觉情感分析模型。通过卷积神经网络提取图像多通道的多层次特征，根据空间注意力机制对多通道的低层特征赋予空间注意力权重，利用通道注意力机制对多通道的高层特征赋予通道注意力权重，分别强化不同层次的特征表示，将强化后的高层特征和低层特征进行融合，形成用于训练情感分类器的判别性特征。在3个真实数据集Twitter Ⅰ、Twitter Ⅱ和EmotionROI上进行对比实验，结果表明，该模型的分类准确率分别达到79.83%、78.25%和49.34%，有效提升了社交媒体视觉情感分析的效果。

关键词: 社交媒体, 视觉情感分析, 卷积神经网络, 注意力, 特征融合

CLC Number:

TP391

CAI Guoyong, CHU Yangyang. Visual Sentiment Analysis Based on Multi-level Features Fusion of Dual Attention[J]. Computer Engineering, 2021, 47(9): 227-234.

蔡国永, 储阳阳. 基于双注意力多层特征融合的视觉情感分析[J]. 计算机工程, 2021, 47(9): 227-234.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0058303

http://www.ecice06.com/EN/Y2021/V47/I9/227

Figures/Tables 11

References

[1] MACHKJDIK J, HANBURY A.Affective image classification using features inspired by psychology and art theory[C]//Proceedings of the 18th ACM International Conference on Multimedia.New York, USA:ACM Press, 2010:83-92.
[2] SIMONYAN K, ZISSERMAN A.Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-10-14].https://arxiv.xilesou.top/pdf/1409.1556.pdf.
[3] 高玮军, 杨杰, 张春霞, 等.基于AT-DPCNN模型的情感分析研究[J].计算机工程, 2020, 46(11):53-60. GAO W J, YANG J, ZHANG C X, et al.Sentiment analysis research based on AT-DPCNN model[J].Computer Engineering, 2020, 46(11):53-60.(in Chinese)
[4] YOU Q, LUO J, JIN H, et al.Robust image sentiment analysis using progressively trained and domain transferred deep networks[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence.Austin, Texas:AAAI Press, 2015:381-388.
[5] YANULEVSKAYA V, VAN GEMERT J C, ROTH K, et al.Emotional valence categorization using holistic image features[C]//Proceedings of International Conference on Image Processing.Piscataway, USA:IEEE Press, 2008:101-104.
[6] SIERSDORFER S, MINACK E, DENG F, et al.Analyzing and predicting sentiment of images on the social Web[C]//Proceedings of the 18th ACM International Conference on Multimedia.New York, USA:ACM Press, 2010:715-718.
[7] ZHAO S, GAO Y, JIANG X, et al.Exploring principles-of-art features for image emotion recognition[C]//Proceedings of the 22nd ACM International Conference on Multimedia.New York, USA:ACM Press, 2014:47-56.
[8] CHEN T, YU F X, CHEN J, et al.Object-based visual sentiment concept analysis and application[C]//Proceedings of the 22nd ACM International Conference on Multimedia.New York, USA:ACM Press, 2014:367-376.
[9] YUAN J, MCDONOUGH S, YOU Q, et al.Sentribute:image sentiment analysis from a mid-level perspective[C]//Proceedings of the 2nd International Workshop on Issues of Sentiment Discovery and Opinion Mining.New York, USA:ACM Press, 2013:1-10.
[10] BORTH D, JI R, CHEN T, et al.Large-scale visual sentiment ontology and detectors using adjective noun pairs[C]//Proceedings of the 21st ACM International Conference on Multimedia.New York, USA:ACM Press, 2013:223-232.
[11] CHEN T, BORTH D, DARRELL T, et al.Deepsentibank:visual sentiment concept classification with deep convolutional neural networks[EB/OL].[2020-04-10].https://arxiv.xilesou.top/pdf/1410.8586.pdf.
[12] PENG K C, CHEN T, SAGOVNIK A, et al.A mixed bag of emotions:model, predict, and transfer emotion distributions[C]//Proceedings of IEEE CVPRʼ15.Piscataway, USA:IEEE Press, 2015:860-868.
[13] YOU Q Z, LUO J B, JIN H L, et al.Building a large scale dataset for image emotion recognition:the fine print and the benchmark[C]//Proceedings of Thirtieth AAAI Conference on Artificial Intelligence.Phoenix, Arizona:AAAI Press, 2016:308-314.
[14] 曹建芳, 陈俊杰, 李海芳.基于Adaboost-BP神经网络的图像情感分类方法研究[J].山西大学学报(自然科学版), 2013, 36(3):331-337. CAO J F, CHEN J J, LI H F.Research on image sentiment classification based on adabost-BP neural network[J].Journal of Shanxi University(Natural Science Edition), 2013, 36(3):331-337.(in Chinese)
[15] YANGJ, SHE D, SUN M, et al.Visual sentiment prediction based on automatic discovery of affective regions[J].IEEE Transactions on Multimedia, 2018, 20(9):2513-2525.
[16] 蔡国永, 吕光瑞, 徐智.基于层次化深度关联融合网络的社交媒体情感分类[J].计算机研究与发展, 2019, 56(6):1312-1324. CAI G Y, LV G R, XU Z.A hierarchical deep correlative fusion network for sentiment classification in social media[J].Journal of Computer Research and Development, 2019, 56(6):1312-1324.(in Chinese)
[17] 王伟凝, 李乐敏, 黄杰雄, 等.基于多层次深度卷积神经网络的图像情感分类[J].华南理工大学学报(自然科学版), 2019, 47(6):39-50. WANG W N, LI L M, HUANG J X, et al.Image emotion classification based on multi-level deep convolution neural network[J].Journal of South China University of Technology(Natural Science Edition), 2019, 47(6):39-50.(in Chinese)
[18] HU J, SHEN L, SUN G.Squeeze-and-excitation networks[C]//Proceedings of IEEE CVPRʼ18.Piscataway, USA:IEEE Press, 2018:7132-7141.
[19] CHEN L, ZHANG H, XIAO J, et al.SCA-CNN:spatial and channel-wise attention in convolutional networks for image captioning[C]//Proceedings of IEEE CVPRʼ17.Piscataway, USA:IEEE Press, 2017:5659-5667.
[20] YOU Q, JIN H, LUO J.Visual sentiment analysis by attending on local image regions[C]//Proceedings of the 31st AAAI Conference on Artificial Intelligence.San Francisco, USA:AAAI Press, 2017:231-237.
[21] SONG K, YAO T, LING Q, et al.Boosting image sentiment analysis with visual attention[J].Neurocomputing, 2018, 312:218-228.
[22] ZEILER M D, FERGUS R.Visualizing and understanding convolutional networks[C]//Proceedings of European Conference on Computer Vision.Berlin, Germany:Springer, 2014:818-833.
[23] PENG K C, SADOVNIK A, GALLAGHER A, et al.Where do emotions come from? predicting the emotion stimuli map[C]//Proceedings of IEEE ICIPʼ16.Piscataway, USA:IEEE Press, 2016:614-618.
[24] CAI G, HE X, PAN J.Visual sentiment analysis with local object regions attention[C]//Proceedings of International Conference on Pioneering Computer Scientists, Engineers and Educators.Singapore:[s.n.], 2019:479-489.

Please choose a citation manager

Content to export