Adversarial Training Based Pseudo Label Constraint Auto-Encoder

doi:10.19678/j.issn.1000-3428.0065936

Abstract

Abstract:

Nodes in social networks often suffer from missing labels, labeling errors, and high manual labeling costs, which negatively affect supervised and semi-supervised network representation learning. In this paper, a self-supervised network representation learning model, the Adversarial Training based Pseudo Label Constraint Auto-Encoder(AT-PLCAE) is proposed, whereby an autoencoder constrained by pseudo-label is designed. By reducing the distance between the pseudo-label of the original graph and that of the network representation, the model reduces information loss during the coding process, thereby learning effectively. Adversarial networks are designed so that the pseudo-label-constrained auto-encoder can organize the potential spatial structure of the representation layer. By forcing the posterior distribution of the latent representation to match the specific prior distribution of the input, the model alleviates the overfitting problem and improves generalization. Node classification experiments on four public datasets: Cora, Citeseer, Wiki, and PubMed, showed that the AT-PLCAE model is superior to the benchmark method. Compared with the benchmark method, AT-PLCAE improved the highest classification accuracy value by 0.018 on Cora, 0.011 on CiteSeer, and PubMed, respectively. The ablation experiment results showed that adversarial networks designed for the pseudo-label-constrained auto-encoder enhance the generalization of the model.

Key words: self-supervised learning, network representation learning, pseudo label, auto-encoder, adversarial training, generalization

摘要：

社会网络的节点经常存在标注缺失、标注错误和人工标注成本高的现象，这种现象对监督或半监督的网络表示学习效果产生了不利的影响。提出一个自监督学习的网络表示学习模型——基于对抗训练的伪标签约束自编码器（AT-PLCAE）。设计一个伪标签约束自编码器，通过缩短原始图的伪标签和网络表示的伪标签之间的距离，减少编码过程中产生的信息损失，约束和引导模型有效学习。同时，设计与伪标签约束自编码器相适应的对抗网络，组织表示的潜在空间结构。将潜在表示后验分布与输入的特定先验分布相匹配后，该模型能够缓解过拟合问题并提升模型的泛化能力。在Cora、Citeseer、Wiki和Pubmed这4个公开数据集上进行节点分类实验，结果表明，AT-PLCAE模型在分类准确率方面学习效果优于基准方法，与基准方法的最高分类准确率相比，在Cora数据集上提升0.018，在Citeseer和Pubmed数据集上均提升0.011。同时消融实验结果表明，针对伪标签约束自编码器的对抗训练增强了模型的泛化能力。

关键词: 自监督学习, 网络表示学习, 伪标签, 自编码器, 对抗训练, 泛化

Kun FU, Minglei SUN, Yuhan HAO, Yinghua LIU. Adversarial Training Based Pseudo Label Constraint Auto-Encoder[J]. Computer Engineering, 2023, 49(11): 123-130.

富坤, 孙明磊, 郝玉涵, 刘赢华. 基于对抗训练的伪标签约束自编码器[J]. 计算机工程, 2023, 49(11): 123-130.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065936

https://www.ecice06.com/EN/Y2023/V49/I11/123

Figures/Tables 8

Fig.1 Model framework

Fig.2 Visualization diagrams of presentation layer of Cora dataset

Fig.3 Infulence of representation hidden layer dimension on algorithm performance

Fig.4 Influence of pseudo label constraint weight on algorithm performance

References 25

1	LIU X Y , TANG J . Network representation learning: a macro and micro view. AI Open, 2021, 2, 43- 64. doi: 10.1016/j.aiopen.2021.02.001
2	HU Z N, DONG Y X, WANG K S, et al. GPT-GNN: generative pre-training of graph neural networks[C]//Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2020: 1857-1867.
3	VELIČKOVIĆ P, FEDUS W, HAMILTON W L, et al. Deep graph infomax[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1809.10341.pdf.
4	ZHU Q K, DU B, YAN P K. Self-supervised training of graph convolutional networks[EB/OL]. [2022-09-10]. https://arxiv.org/abs/2006.02380.pdf.
5	LIU Y X , JIN M , PAN S R , et al. Graph self-supervised learning: a survey. IEEE Transactions on Knowledge and Data Engineering, 2023, 35 (6): 5879- 5900.
6	MAKHZANI A, SHLENS J, JAITLY N, et al. Adversarial autoencoders[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1511.05644.pdf.
7	DONAHUE J, KRÄHENBÜHL P, DARRELL T. Adversarial feature learning[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1605.09782.pdf.
8	DUMOULIN V, BELGHAZI I, POOLE B, et al. Adversarially learned inference[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1606.00704.pdf.
9	PAN S R , HU R Q , FUNG S F , et al. Learning graph embedding with adversarial training methods. IEEE Transactions on Cybernetics, 2020, 50 (6): 2475- 2487. doi: 10.1109/TCYB.2019.2932096
10	YOU Y N, CHEN T L, WANG Z Y, et al. When does self-supervision help graph convolutional networks?[EB/OL]. [2022-09-10]. https://arxiv.org/abs/2006.09136.pdf.
11	WANG C, PAN S R, LONG G D, et al. MGAE: marginalized graph autoencoder for graph clustering[C]//Proceedings of 2017 ACM Conference on Information and Knowledge Management. New York, USA: ACM Press, 2017: 889-898.
12	PARK J, LEE M, CHANG H J, et al. Symmetric graph convolutional autoencoder for unsupervised graph representation learning[C]//Proceedings of IEEE/CVF International Conference on Computer Vision. Washington D.C., USA: IEEE Press, 2020: 6518-6527.
13	KIPF T N, WELLING M. Semi-supervised classification with graph convolutional networks[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1609.02907.pdf.
14	WANG D X, CUI P, ZHU W W. Structural deep network embedding[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2016: 1225-1234.
15	KIPF T N, WELLING M. Variational graph auto-encoders[EB/OL]. [EB/OL]. [2022-09-10]. https://arxiv.org/abs/1611.07308.pdf.
16	VELIČKOVIĆ P, CUCURULL G, CASANOVA A, et al. Graph attention networks[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1710.10903.pdf.
17	XU K, HU W H, LESKOVEC J, et al. How powerful are graph neural networks?[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1810.00826.pdf.
18	KINGMA D P, WELLING M. Auto-encoding variational Bayes[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1312.6114.pdf.
19	富坤, 赵晓梦, 付紫桐, 等. 基于不完全信息的深度网络表示学习方法. 计算机科学, 2021, 48 (12): 212- 218. URL
	FU K , ZHAO X M , FU Z T , et al. Deep network representation learning method on incomplete information networks. Computer Science, 2021, 48 (12): 212- 218. URL
20	CAO S S, LU W, XU Q K. Deep neural networks for learning graph representations[C]//Proceedings of the 30th AAAI Conference on Artificial Intelligence. New York, USA: ACM Press, 2016: 1145-1152.
21	GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2014: 2672-2680.
22	ARJOVSKY M, CHINTALA S, BOTTOU L. Wasserstein GAN[EB/OL]. [2022-09-10]. https://arxiv.org/abs/1701.07875.pdf.
23	PEROZZI B, AL-RFOU R, SKIENA S. DeepWalk: online learning of social representations[C]//Proceedings of the 20th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2014: 701-710.
24	GROVER A, LESKOVEC J. node2vec: scalable feature learning for networks[C]//Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York, USA: ACM Press, 2016: 855-864.
25	HUANG X A, LI J D, HU X A. Accelerated attributed network embedding[C]//Proceedings of 2017 SIAM International Conference on Data Mining. Philadelphia, USA: Society for Industrial and Applied Mathematics, 2017: 633-641.

[1]	WANG Yanguo, LÜ Pengyuan, LAN Jinjiang, LIU Mingzhe, QIN Guanjun, ZHANG Shuohua, ZHOU Yu. Wind Turbine Fault Classification Method Based on Adversarial Training and Transformer [J]. Computer Engineering, 2024, 50(9): 377-384.
[2]	Zhengkang ZHANG, Dan YANG, Tiezheng NIE, Yue KOU. Self-Supervised Learning Based on Graph Structural Clustering for Disease Diagnosis Method [J]. Computer Engineering, 2024, 50(7): 360-371.
[3]	WEI Zhuoyi, LUO Mai, LI Wenbing, ZENG Yuansong, YU Weijiang, YANG Yuedong. Intelligent Single-Cell Classification Based on Multisource Domain Adaptation [J]. Computer Engineering, 2024, 50(6): 48-55.
[4]	YU Changhong, XU Konghao, ZHANG Ze, GAO Ming. Improving Network Intrusion Detection Methods in Isolated Forests Based on Split Points [J]. Computer Engineering, 2024, 50(6): 148-156.
[5]	LI Yongfei, LI Mingyang, CHANG Xin, CAO Kexin. Anomaly Detection of IoT Water Quality Monitoring Data Based on Explainable Deep Learning [J]. Computer Engineering, 2024, 50(6): 179-187.
[6]	LI Jing, LI Jian, CHEN Haifeng, ZHANG Qian, WANG Liyan, PEI Ercheng. Facial Expression Recognition Based on Key Region Masking and Reconstruction [J]. Computer Engineering, 2024, 50(5): 241-249.
[7]	ZHOU Zhaochen, FANG Qingmao, WU Xiaohong, HU Ping, HE Xiaohai. Machine Reading Comprehension Model Based on MacBERT and Adversarial Training [J]. Computer Engineering, 2024, 50(5): 41-50.
[8]	WANG Lin, HUANG Hao. Voice Conversion Combining Vector Quantization and CTC Introducing Pre-Trained Representation [J]. Computer Engineering, 2024, 50(4): 313-320.
[9]	Lin CHEN, Yuankai FAN, Zhenying HE, Xiaoqing LIU, Yang YANG, Lumin TANG. Combinatorial Generalization Ability Evaluation Method of SQL-to-text Model [J]. Computer Engineering, 2024, 50(3): 326-335.
[10]	REN Yi, SU Bo, YUAN Shuai. Multidimensional Feature Named Entity Recognition Method in Education Domain [J]. Computer Engineering, 2024, 50(10): 110-118.
[11]	DAI Lei, CAO Lin, GUO Yanan, ZHANG Fan, DU Kangning. Deepfake Cross-Model Defense Method Based on Generative Adversarial Network [J]. Computer Engineering, 2024, 50(10): 100-109.
[12]	DONG Fengkai, ZOU Xiaoqiang, WANG Jiahui, MA Liming, YANG Wenyuan, LIU Xiyao. Dual-Stream Generalized Face Forgery Detection Method Based on Intra-Inter Frame Self-Blending [J]. Computer Engineering, 2024, 50(10): 185-195.
[13]	Zhaoxian XIE, Xingmin ZOU, Wenjing ZHANG. High-Efficient Parameter-Pruning Algorithm of Decision Tree for Large Dataset [J]. Computer Engineering, 2024, 50(1): 156-165.
[14]	Zhidong SHEN, Hengxian YUE. Textual Adversarial Training Method Based on Distributed Perturbation [J]. Computer Engineering, 2023, 49(9): 16-22.
[15]	Xiaoli LIU, Yitong WANG. Multi-density Graph-based Session Recommendation Using Self-supervised Learning [J]. Computer Engineering, 2023, 49(9): 60-68, 78.

Please choose a citation manager

Content to export