Semi-Supervised Text Classification Method Based on Multi-Granularity Semantic Margin Loss

doi:10.19678/j.issn.1000-3428.0070533

Abstract

Abstract:

Semi-supervised text classification uses a limited set of labeled data in conjunction with a large corpus of unlabeled data to develop text classification models. Current pseudo-labeling techniques often face challenges such as pseudo-label bias stemming from decision boundary underfitting and error accumulation. To address these issues, a cross-annotation model based on the multi-granularity semantic margin loss is introduced. Initially, the Bidirectional Encoder Representations from Transformers (BERT)-pretrained model is employed to extract vector sequences enriched with contextual information. Subsequently, an attention mechanism and Text Convolutional Neural Network (TextCNN) are utilized to capture global and local semantic features, respectively. These features are subsequently fused to create enriched semantic representations, factoring in semantic similarities across different categories to separate similar classes within the feature-embedding space, thereby reducing the impact of error accumulation. Additionally, a category-specific margin to calculate the positive and negative sample loss between pseudo-labels derived through adaptive local threshold learning is introduced. The margin enables a high-density distribution of samples within each class in the embedding space, thereby mitigating decision boundary underfitting. Experimental evaluations demonstrate that the proposed model effectively integrates multi-granular semantic information, achieves high cohesion among pseudo-labeled samples, and significantly improves the performance of semi-supervised text classification.

Key words: text classification, pseudo-label, margin loss, feature extraction, decision boundary, semi-supervised learning, natural language processing

摘要：

半监督文本分类使用少量标注数据和大量未标注数据训练文本分类模型。然而, 现有基于伪标签的方法存在由于决策边界欠拟合和伪标签偏差导致的误差累积问题。为此, 本文提出一个基于多粒度语义间隔损失的交叉标注模型。首先通过基于Transformer的双向编码器表示(BERT)预训练模型提取包含上下文信息的向量序列, 然后利用注意力网络和文本卷积神经网络(TextCNN)分别捕获全局语义特征和局部语义特征, 并将这两部分特征融合形成新的语义特征, 同时考虑不同类别之间的语义相似性, 以在特征嵌入空间中将相似类别的样本分开, 从而减轻误差累积的影响。最后, 增加一个与类别相关的间隔, 通过自适应局部阈值学习得到的伪标签样本与间隔进行正负样本损失计算, 使每个类别样本在嵌入空间形成高密度分布, 从而缓解决策边界的欠拟合。实验结果表明, 该模型能够有效融合多粒度语义信息, 生成高内聚伪标签样本, 提升半监督文本分类的性能。

关键词: 文本分类, 伪标签, 间隔损失, 特征提取, 决策边界, 半监督学习, 自然语言处理

LIAO Guoan, XU Ji, CHEN Yanping. Semi-Supervised Text Classification Method Based on Multi-Granularity Semantic Margin Loss[J]. Computer Engineering, 2026, 52(7): 107-120.

廖国安, 徐计, 陈艳平. 基于多粒度语义间隔损失的半监督文本分类方法[J]. 计算机工程, 2026, 52(7): 107-120.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0070533

https://www.ecice06.com/EN/Y2026/V52/I7/107

Figures/Tables 14

Fig.1 Illustration of the overall model

Fig.2 Illustration of the margin loss

Fig.3 Accuracy comparison under different numbers of labels

Fig.4 GoEmo training set and accuracy heat map

Fig.5 Illustration of accuracy among multi-class datasets

References 41

1	张铭泉, 周辉, 曹锦纲. 基于注意力机制的双BERT有向情感文本分类研究. 智能系统学报, 2022, 17(6): 1220- 1227. doi: 10.3969/j.issn.1008-0775.2017.11.002
	ZHANG M Q, ZHOU H, CAO J G. Dual BERT directed sentiment text classification based on attention mechanism. CAAI Transactions on Intelligent Systems, 2022, 17(6): 1220- 1227. doi: 10.3969/j.issn.1008-0775.2017.11.002
2	古丽娜孜·艾力木江, 乎西旦·居马洪, 孙铁利, 等. 基于支持向量的最近邻文本分类方法. 智能系统学报, 2018, 13(5): 799- 807. doi: 10.11992/tis.201711007
	Gulnaz Alimjan, Hurxida Jumahun, SUN T L, et al. The nearest neighbor text classification method based on support vector. CAAI Transactions on Intelligent Systems, 2018, 13(5): 799- 807. doi: 10.11992/tis.201711007
3	SOSEA T, CARAGEA C. Leveraging training dynamics and self-training for text classification[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022: 4750-4762.
4	XIE Q Z, DAI Z H, HOVY E, et al. Unsupervised data augmentation for consistency training. Advances in Neural Information Processing Systems, 2020, 33, 6256- 6268. doi: 10.48550/arXiv.1904.12848
5	武红鑫, 韩萌, 陈志强, 等. 监督和半监督学习下的多标签分类综述. 计算机科学, 2022, 49(8): 12- 25.
	WU H X, HAN M, CHEN Z Q, et al. Survey of multi-label classification based on supervised and semi-supervised learning. Computer Science, 2022, 49(8): 12- 25.
6	NOVOTNEY S, SCHWARTZ R, KHUDANPUR S. Getting more from automatic transcripts for semi-supervised language modeling. Computer Speech & Language, 2016, 36, 93- 109. doi: 10.1016/j.csl.2015.08.007
7	LEE J H, KO S K, HAN Y S. SALNet: semi-supervised few-shot text classification with attention-based lexicon construction[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l.]: AAAI, 2021: 13189-13197.
8	SAJJADI M, JAVANMARDI M, TASDIZEN T. Regularization with stochastic transformations and perturbations for deep semi-supervised learning[EB/OL]. [2024-09-24]. https://arxiv.org/abs/1606.04586.
9	YIN J, LIU X Y, YANG Z W. A deep multiple-instance text binary classification for topic relevant content extraction on social media. Journal of King Saud University-Computer and Information Sciences, 2024, 36(1): 101883. doi: 10.1016/j.jksuci.2023.101883
10	GIDARIS S, KOMODAKIS N. Dynamic few-shot visual learning without forgetting[C]// Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE Press, 2018: 4367-4375.
11	HUANG Y G, WANG Y H, TAI Y, et al. CurricularFace: adaptive curriculum learning loss for deep face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Seattle, USA: IEEE Press, 2020: 5900-5909.
12	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2024-09-24]. https://arxiv.org/abs/1810.04805.
13	LAI S W, XU L H, LIU K, et al. Recurrent convolutional neural networks for text classification[C]//Proceedings of the 29th AAAI Conference on Artificial Intelligence. [S. l.]: AAAI, 2015: 2267-2273.
14	SOHN K, BERTHELOT D, LI C L, et al. FixMatch: simplifying semi-supervised learning with consistency and confidence[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2001.07685.
15	CHEN H, HAN W, PORIA S. SAT: improving semi-supervised text classification with simple instance-adaptive self-training[C]//Proceedings of the Findings of the Association for Computational Linguistics: EMNLP 2022. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics, 2022: 6141-6146.
16	CHEN J A, YANG Z C, YANG D Y. MixText: linguistically-informed interpolation of hidden space for semi-supervised text classification[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 2147-2157.
17	ZHANG H Y, CISSE M, DAUPHIN Y N, et al. Mixup: beyond empirical risk minimization[EB/OL]. [2024-09-24]. https://arxiv.org/abs/1710.09412.
18	ZOU H P, ZHOU Y, CARAGEA C, et al. CrisisMatch: semi-supervised few-shot learning for fine-grained disaster tweet classification[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2310.14627.
19	YANG W Y, ZHANG R C, CHEN J F, et al. Prototype-guided pseudo labeling for semi-supervised text classification[C]//Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). Toronto, Canada: Association for Computational Linguistics, 2023: 16369-16382.
20	ZHANG B W, WANG Y D, HOU W X, et al. FlexMatch: boosting semi-supervised learning with curriculum pseudo labeling[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2110.08263.
21	WANG Y D, CHEN H, HENG Q, et al. FreeMatch: self-adaptive thresholding for semi-supervised learning[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2205.07246.
22	BLUM A, MITCHELL T. Combining labeled and unlabeled data with co-training[C]//Proceedings of the 11th Annual Conference on Computational Learning Theory. New York, USA: ACM Press, 1998: 92-100.
23	ZOU H P, CARAGEA C. JointMatch: a unified approach for diverse and collaborative pseudo-labeling to semi-supervised text classification[C]//Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing. Singapore: Association for Computational Linguistics, 2023: 7290-7301.
24	MNIH V, HEESS N M O, GRAVES A, et al. Recurrent models of visual attention[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2014: 2204-2212.
25	VASWANI A. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
26	FLORIDI L, CHIRIATTI M. GPT-3: its nature, scope, limits, and consequences. Minds and Machines, 2020, 30(4): 681- 694. doi: 10.1007/s11023-020-09548-1
27	李永忠, 郑滔. 基于标签的半监督HDP文本分类主题模型. 模式识别与人工智能, 2017, 30(12): 1138- 1148. doi: 10.16451/j.cnki.issn1003-6059.201712010
	LI Y Z, ZHENG T. Semi-supervised labeled hierarchical dirichlet process topic model for document categorization. Pattern Recognition and Artificial Intelligence, 2017, 30(12): 1138- 1148. doi: 10.16451/j.cnki.issn1003-6059.201712010
28	林呈宇, 王雷, 薛聪. 标签语义增强的弱监督文本分类模型. 计算机应用, 2023, 43(2): 335- 342.
	LIN C Y, WANG L, XUE C. Weakly-supervised text classification with label semantic enhancement. Journal of Computer Applications, 2023, 43(2): 335- 342.
29	LIN Y C, LI J, XIAO H, et al. Automatic literature screening using the PAJO deep-learning model for clinical practice guidelines. BMC Medical Informatics and Decision Making, 2023, 23(1): 247. doi: 10.1186/s12911-023-02328-8
30	DENG J K, GUO J, XUE N N, et al. ArcFace: additive angular margin loss for deep face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Long Beach, USA: IEEE Press, 2020: 4685-4694.
31	WANG H, WANG Y T, ZHOU Z, et al. CosFace: large margin cosine loss for deep face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Salt Lake City, USA: IEEE Press, 2018: 5265-5274.
32	KIM M, JAIN A K, LIU X M. AdaFace: quality adaptive margin for face recognition[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). New Orleans, USA: IEEE Press, 2022: 18729-18738.
33	HOSSEINI M, CARAGEA C. Semi-supervised domain adaptation for emotion-related tasks[C]//Proceedings of the Findings of the Association for Computational Linguistics: ACL 2023. Toronto, Canada: Association for Computational Linguistics, 2023: 5402-5410.
34	XU H M, LIU L Q, ABBASNEJAD E. Progressive class semantic matching for semi-supervised text classification[C]//Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Seattle, USA: Association for Computational Linguistics, 2022: 3003-3013.
35	ZHANG X, ZHAO J B, LECUN Y. Character-level convolutional networks for text classification[EB/OL]. [2024-09-24]. https://arxiv.org/abs/1509.01626.
36	CHANG M W, RATINOV L A, ROTH D, et al. Importance of semantic representation: dataless classification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. [S. l.]: AAAI, 2008: 830-835.
37	MAAS A L, DALY R E, PHAM P T, et al. Learning word vectors for sentiment analysis[C]//Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies. Portland, USA: Association for Computational Linguistics, 2011: 142-150.
38	RASHKIN H, SMITH E M, LI M, et al. Towards empathetic open-domain conversation models: a new benchmark and dataset[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics. Florence, Italy: Association for Computational Linguistics, 2019: 5370-5381.
39	DEMSZKY D, MOVSHOVITZ-ATTIAS D, KO J, et al. GoEmotions: a dataset of fine-grained emotions[C]//Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. Online: Association for Computational Linguistics, 2020: 4040-4054.
40	ALAM F, QAZI U, IMRAN M, et al. HumAID: human-annotated disaster incidents data from Twitter with deep learning benchmarks[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2104.03090.
41	CHEN H, TAO R, FAN Y, et al. SoftMatch: addressing the quantity-quality trade-off in semi-supervised learning[EB/OL]. [2024-09-24]. https://arxiv.org/abs/2301.10921.

[1]	SUN Yu, WANG Hongjie, DU Yanhui, LIU Nan. Multi-Level Chinese Adversarial Example Generation Method Based on Glyph and Semantic [J]. Computer Engineering, 2026, 52(7): 287-301.
[2]	YANG Xinyi, HE Xiaohai, TENG Qizhi, CHEN Honggang, QING Linbo, WU Xiaohong. Cross-Modal Named Entity Recognition Based on Semantic Fusion [J]. Computer Engineering, 2026, 52(7): 345-353.
[3]	DANG Weichao, PEI Lixian, GAO Gaimei, LIU Chunxia. Weakly-Supervised Temporal Action Localization via Video—Text Semantic Alignment and Full Video Dependency [J]. Computer Engineering, 2026, 52(7): 354-367.
[4]	LIU Yujie, DU Zhonghao, LI Xuanting, LI Zongmin. Semi-supervised Medical Image Segmentation Model Based on Selective Supervision and Dynamic Threshold [J]. Computer Engineering, 2026, 52(6): 121-131.
[5]	LI Xuexiang, ZHENG Yongli, ZHANG Yize, DUAN Pengsong. Review of Traffic Analysis Methods Based on Machine Learning and Pre-trained Model [J]. Computer Engineering, 2026, 52(6): 53-67.
[6]	XU Minchen, QU Dan, SI Nianwen, PENG Sisi, CHEN Yaqi. Technologies for Detecting Disinformation in Social Media: A Comprehensive Review [J]. Computer Engineering, 2026, 52(5): 60-80.
[7]	LI Luyang, YAN Jinlong, FANG Zeru, JIN Qiqi, XUE Hongxin. 3D Small Object Detection Algorithm Based on Dynamic Feature Enhancement [J]. Computer Engineering, 2026, 52(4): 264-275.
[8]	CHEN Zhenqing, WAN Jiafu, ZHANG Rui. Adaptive Lossless Segmented Compression Method Integrating Temporal Dependencies and Data Features [J]. Computer Engineering, 2026, 52(2): 79-88.
[9]	LIU Chang, LIANG Bingxue, TIAN Rongkun, QIN Yuhua. Medical and Health Question Classification Based on Multi-feature Fusion and Hybrid Neural Network [J]. Computer Engineering, 2026, 52(2): 342-355.
[10]	ZHANG Chenghui, LUO Jing, TU Xinhui, CHEN Yulin. Automatic Corpus Query Generation Method Based on Large Language Model [J]. Computer Engineering, 2026, 52(2): 404-412.
[11]	DAN Chonghong, WEI Honglei, HE Zhou, WU Guanfeng. SRMpose: Multi-Scale Feature Extraction Keypoint Detection Algorithm [J]. Computer Engineering, 2026, 52(2): 136-147.
[12]	RAN Tongyu, TANG Mengzi, XIE Qing, LIU Yongjian. Semi-Supervised Ordinal Classification Framework Based on Information Fusion [J]. Computer Engineering, 2026, 52(2): 287-298.
[13]	CHEN Yumin, CHE Jin, YANG Yingying. Unsupervised Person Re-Identification for Multigrained Teacher-Student Networks Incorporating Spatial Frequency Information [J]. Computer Engineering, 2026, 52(1): 217-227.
[14]	WANG Chundong, ZHAO Zhihang, YANG Weijie, FANG Shunyao. Adversarial Sample Generation Method for Chinese Text Based on Word Reproduction [J]. Computer Engineering, 2026, 52(1): 303-313.
[15]	ZHU Siyuan, LI Jiasheng, ZOU Danping, HE Di, YU Wenxian. Unstructured Road Defect Detection Algorithm Based on Semi-Supervised Learning [J]. Computer Engineering, 2025, 51(9): 14-24.

Please choose a citation manager

Content to export