Depression Detection Based on Contextual Knowledge Enhanced Transformer Network

doi:10.19678/j.issn.1000-3428.0067936

Abstract

Abstract:

Depression, as a prevalent mental health problem, substantially impacts individual′s daily lives and well-being. Addressing the limitations of current depression detection, such as subjectivity and manual intervention, automatic detection methods based on deep learning have become a popular research direction. The primary challenge in the most accessible text modality is modelling the long-range and sequence dependencies in depressive texts. To address this problem, this paper proposes a contextual knowledge-based enhanced Transformer network model, named Robustly optimized Bidirectional Encoder Representations from Transformers approach-Bidirectional Long Short-Term Memory(RoBERTa-BiLSTM), to comprehensively extract and utilize contextual features from depressive text sequences. By combining the strengths of sequence models and Transformer architectures, the proposed model captures contextual interactions between words to provide a reference for depression category prediction and information characterization. First, the RoBERTa model is employed to embed vocabulary into a semantic vector space, and then, a BiLSTM network effectively captures long-range contextual semantics. Finally, empirical research is conducted on two large-scale datasets, DAIC-WOZ and EATD-Corpus. Experimental results demonstrate that the model achieves an accuracy exceeding 0.74 and 0.93, and a recall exceeding 0.66 and 0.56, respectively, enabling accurate depression detection.

Key words: depression detection, sequence model, deep learning, Transformer model, Bi-directional Long Short-Term Memory(BiLSTM) model

摘要：

抑郁症作为一种常见的心理健康问题，严重影响人们的日常生活甚至是生命安全。鉴于目前的抑郁症检测存在主观性和人工干预等缺点，基于深度学习的自动检测方式成为热门研究方向。对于最易获取的文本模态而言，主要的挑战在于如何建模抑郁文本中的长距离依赖与序列依赖。为解决该问题，提出一种基于上下文知识的增强型Transformer网络模型RoBERTa-BiLSTM，旨在从抑郁文本序列中充分提取和利用上下文特征。结合序列模型与Transformer模型优点，建模单词间上下文交互，为抑郁类别揭示与信息表征提供参考。首先，利用RoBERTa方法将词汇嵌入到语义向量空间；其次，利用双向长短期记忆网络(BiLSTM)模型有效捕获长距离上下文语义；最后，在DAIC-WOZ和EATD-Corpus 2个大规模数据集上进行实证研究。实验结果显示，RoBERTa-BiLSTM模型的准确率分别达到0.74和0.93以上，召回率分别达到0.66和0.56以上，能够准确地检测抑郁症。

关键词: 抑郁检测, 序列模型, 深度学习, Transformer模型, 双向长短期记忆模型

Yazhou ZHANG, Yu HE, Lu RONG, Xiangkai WANG. Depression Detection Based on Contextual Knowledge Enhanced Transformer Network[J]. Computer Engineering, 2024, 50(8): 75-85.

张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067936

https://www.ecice06.com/EN/Y2024/V50/I8/75

Figures/Tables 13

Fig.1 Structure of the RoBERTa-BiLSTM model

Fig.2 Structure of the Transformer model

Fig.3 Structure of the LSTM network unit

Fig.4 Structure of the BiLSTM network

Fig.5 Loss changes with Epoch

Fig.6 Results of ablation experiment on DAIC-WOZ dataset

Fig.7 Results of ablation experiment on EATD-Corpus dataset

References 28

1	WEI P C, PENG K Y, ROITBERG A, et al. Multi-modal depression estimation based on sub-attentional fusion[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2023: 623-639.
2	FALAGAS M E , VARDAKAS K Z , VERGIDIS P I . Under-diagnosis of common chronic diseases: prevalence and impact on human health. International Journal of Clinical Practice, 2007, 61 (9): 1569- 1579. doi: 10.1111/j.1742-1241.2007.01423.x
3	ZHOU X Z , JIN K , SHANG Y Y , et al. Visually interpretable representation learning for depression recognition from facial images. IEEE Transactions on Affective Computing, 2020, 11 (3): 542- 552. doi: 10.1109/TAFFC.2018.2828819
4	TSAKALIDIS A, LIAKATA M, DAMOULAS T, et al. Can we assess mental health through social media and smart devices? Addressing bias in methodology and evaluation[C]//Proceedings of Joint European Conference on Machine Learning and Knowledge Discovery in Databases. Berlin, Germany: Springer, 2019: 407-423.
5	王款, 宣士斌, 何雪东, 等. 基于交叉注意力Transformer的人体姿态估计方法. 计算机工程, 2023, 49 (7): 223- 231. URL
	WANG K , XUAN S B , HE X D , et al. Human pose estimation method based on cross attention Transformer. Computer Engineering, 2023, 49 (7): 223- 231. URL
6	LIU Z, LIN W, SHI Y, et al. A robustly optimized BERT pre-training approach with post-training[C]//Proceedings of China National Conference on Chinese Computational Linguistics. Berlin, Germany: Springer, 2021: 471-484.
7	张应成, 杨洋, 蒋瑞, 等. 基于BiLSTM-CRF的商情实体识别模型. 计算机工程, 2019, 45 (5): 308- 314. URL
	ZHANG Y C , YANG Y , JIANG R , et al. Commercial intelligence entity recognition model based on BiLSTM-CRF. Computer Engineering, 2019, 45 (5): 308- 314. URL
8	LOSADA D E , GAMALLO P . Evaluating and improving lexical resources for detecting signs of depression in text. Language Resources and Evaluation, 2020, 54 (1): 1- 24. doi: 10.1007/s10579-018-9423-1
9	WOLOHAN J T, HIRAGA M, MUKHERJEE A, et al. Detecting linguistic traces of depression in topic-restricted text: attending to self-stigmatized depression with NLP[C]//Proceedings of the 1st International Workshop on Language Cognition and Computational Models. Stroudsburg, USA: ACL Press, 2018: 11-21.
10	WILLIAMSON J R, GODOY E, CHA M, et al. Detecting depression using vocal, facial and semantic communication cues[C]//Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. New York, USA: ACM Press, 2016: 11-18.
11	YANG L, JIANG D M, HE L, et al. Decision tree based depression classification from audio video and language information[C]//Proceedings of the 6th International Workshop on Audio/Visual Emotion Challenge. New York, USA: ACM Press, 2016: 89-96.
12	LAM G, HUANG D Y, LIN W S. Context-aware deep learning for multi-modal depression detection[C]// Proceedings of the 2019 IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA: IEEE Press, 2019: 3946-3950.
13	韩虎, 赵启涛, 孙天岳, 等. 面向社交媒体评论的上下文语境讽刺检测模型. 计算机工程, 2021, 47 (1): 66- 71. URL
	HAN H , ZHAO Q T , SUN T Y , et al. Contextual sarcasm detection model for social media comments. Computer Engineering, 2021, 47 (1): 66- 71. URL
14	VASHA Z N , SHARMA B , ESHA I J , et al. Depression detection in social media comments data using machine learning algorithms. Bulletin of Electrical Engineering and Informatics, 2023, 12 (2): 987- 996. doi: 10.11591/eei.v12i2.4182
15	TROTZEK M , KOITKA S , FRIEDRICH C M . Utilizing neural networks and linguistic metadata for early detection of depression indications in text sequences. IEEE Transactions on Knowledge and Data Engineering, 2018, 32 (3): 588- 601.
16	AMANAT A , RIZWAN M , JAVED A R , et al. Deep learning for depression detection from textual data. Electronics, 2022, 11 (5): 2079- 9292.
17	YU L X , JIANG W Y , REN Z H , et al. Detecting changes in attitudes toward depression on Chinese social media: a text analysis. Journal of Affective Disorders, 2021, 280 (Pt A): 354- 363.
18	LI Z P , AN Z Y , CHENG W C , et al. MHA: a multimodal hierarchical attention model for depression detection in social media. Health Information Science and Systems, 2023, 11 (1): 6.
19	YADAV U , SHARMA A K . A novel automated depression detection technique using text transcript. International Journal of Imaging Systems and Technology, 2023, 33 (1): 108- 122.
20	SENN S, TLACHAC M L, FLORES R, et al. Ensembles of BERT for depression classification[C]// Proceedings of the 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society. Washington D. C., USA: IEEE Press, 2022: 4691-4694.
21	PENNINGTON J, SOCHER R, MANNING C D. GloVe: global vectors for word representation[C]//Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: ACL Press, 2014: 1532-1543.
22	PETERS M E, NEUMANN M, IYYER M, et al. Deep contextualized word representations[C]//Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: ACL Press, 2018: 2227-2237.
23	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional Transformers for language understanding[C]//Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Stroudsburg, USA: ACL Press, 2019: 4171-4186.
24	贺小伟, 徐靖杰, 王宾, 等. 基于GRU-LSTM组合模型的云计算资源负载预测研究. 计算机工程, 2022, 48 (5): 11-17, 34. URL
	HE X W , XU J J , WANG B , et al. Research on cloud computing resource load forecasting based on GRU-LSTM combination model. Computer Engineering, 2022, 48 (5): 11-17, 34. URL
25	GRATCH J, ARTSTEIN R, LUCAS G, et al. The distress analysis interview corpus of human and computer interviews[C]//Proceedings of the 9th International Conference on Language Resources and Evaluation. Paris, France: ELRA Press, 2014: 3123-3128.
26	SHEN Y, YANG H Y, LIN L. Automatic depression detection: an emotional audio-textual corpus and a GRU/BiLSTM-based model[C]//Proceedings of the 2022 IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA: IEEE Press, 2022: 6247-6251.
27	LIU X Y , WU J X , ZHOU Z H . Exploratory undersampling for class-imbalance learning. IEEE Transactions on Systems, Man, and Cybernetics Part B, Cybernetics, 2009, 39 (2): 539- 550.
28	ALHANAI T, GHASSEMI M, GLASS J. Detecting depression with audio/text sequence modeling of interviews[C]//Proceedings of the Annual Conference of the International Speech Communication Association. Baixas, France: ISCA Press, 2018: 1716-1720.

[1]	ZHU Kai, LI Li, ZHANG Tong, JIANG Sheng, BIE Yiming. Multi-Stage Motion Blur Image Restoration Network Based on Transformer [J]. Computer Engineering, 2024, 50(9): 276-285.
[2]	WEI Wei, DING Xiangxiang, GUO Mengxing, YANG Zhao, LIU Hui. Review of Text Similarity Calculation Methods [J]. Computer Engineering, 2024, 50(9): 18-32.
[3]	ZHANG Tianpeng, HAN Jing, LÜ Xueqiang. Super-Resolution-Aided Small-Target Detection Based on Multi-Task Learning [J]. Computer Engineering, 2024, 50(9): 304-312.
[4]	GAO Yubao, WEN Zhicheng. Dual Decoder Image Denoising Method Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(9): 324-332.
[5]	WANG Yanguo, LÜ Pengyuan, LAN Jinjiang, LIU Mingzhe, QIN Guanjun, ZHANG Shuohua, ZHOU Yu. Wind Turbine Fault Classification Method Based on Adversarial Training and Transformer [J]. Computer Engineering, 2024, 50(9): 377-384.
[6]	Huaqing ZHANG, Zhangtao XIA, Xiaoqing LU, Jijun TONG. Named Entity Recognition of Vascular Surgery Based on Glyph Features [J]. Computer Engineering, 2024, 50(8): 13-21.
[7]	Xuebing BAI, Jin CHE, Jinman WU, Yumin CHEN. Image Captioning Method Based on Transformer Visual Features Fusion [J]. Computer Engineering, 2024, 50(8): 229-238.
[8]	Hong ZHAO, Xiao WANG. Study on Lesion Segmentation of Melanoma Images Based on Swin-Transformer [J]. Computer Engineering, 2024, 50(8): 249-258.
[9]	Wei GAO, Shuailong LI, Lin MAO, Lei WANG, Yingying LI, Lin HAN. One Acceleration Strategy for Operator Generation Based on TVM [J]. Computer Engineering, 2024, 50(8): 353-362.
[10]	Yu WANG, Qi QI, Chun WANG, Cai XU. High-Precision Fault Diagnosis Method for Energy Storage Inverter Signals [J]. Computer Engineering, 2024, 50(8): 389-396.
[11]	Ci XIAO, Yang XU, Yongdan ZHANG, Mingwen FENG, Yiqian HUANG. Nighttime Semantic Segmentation with Attention and Low-Light Enhancement [J]. Computer Engineering, 2024, 50(7): 271-281.
[12]	Shijing ZHANG, Xutao MO, Xing ZHAO, Yanglin DONG. Internal Thread Defect Detection Based on Spherical Catadioptric Imaging and YOLOv7 [J]. Computer Engineering, 2024, 50(7): 282-292.
[13]	Ruiting NUI, Tianfeng YAN, Rui GAO, Yingzhi WANG. Deep Learning TCNN-MobileNet-Based Modulation Recognition Under Low Signal-to-Noise Radio [J]. Computer Engineering, 2024, 50(7): 204-215.
[14]	LI Yongfei, LI Mingyang, CHANG Xin, CAO Kexin. Anomaly Detection of IoT Water Quality Monitoring Data Based on Explainable Deep Learning [J]. Computer Engineering, 2024, 50(6): 179-187.
[15]	LI Zhijin, TANG Jiahui, YAN Jinfeng. Lightweight Recognition Method Based on Edge Computing [J]. Computer Engineering, 2024, 50(6): 287-295.

Please choose a citation manager

Content to export