基于机器学习的用户与实体行为分析技术综述

doi:10.19678/j.issn.1000-3428.0062623

摘要/Abstract

摘要： 随着网络安全技术的更新迭代，新型攻击手段日益增加，企业面临未知威胁难以识别的问题。用户与实体行为分析是识别用户和实体行为中潜在威胁事件的一种异常检测技术，广泛应用于企业内部威胁分析和外部入侵检测等任务。基于机器学习方法对用户和实体的行为进行模型建立与风险点识别，可以有效解决未知威胁难以检测的问题，增强企业网络安全防护能力。回顾用户与实体行为分析的发展历程，重点讨论用户与实体行为分析技术在统计学习、深度学习、强化学习等3个方面的应用情况，研究具有代表性的用户与实体行为分析算法并对算法性能进行对比分析。介绍4种常用的公共数据集及特征工程方法，总结两种增强行为表述准确性的特征处理方式。在此基础上，阐述归纳典型异常检测算法的优劣势，指出内部威胁分析与外部入侵检测的局限性，并对用户与实体行为分析技术未来的发展方向进行展望。

关键词: 网络安全, 用户与实体行为分析, 异常检测, 统计学习, 深度学习, 强化学习

Abstract: With the continuous development of network security technology, new attacking methods are becoming increasingly numerous, exposing enterprises to unknown threats that are difficult to identify.User Entity Behavior Analytics(UEBA) is an anomaly detection technology to identify potential threat events in user and entity behavior.It has been widely used in external intrusion detection and internal threat analysis of enterprises.By using machine learning methods to model user and entity behavior and identify risk points, UEBA can address unknown threats that are difficult to detect, and enhance the defense of enterprise networks.This paper introduces the development history of UEBA, and discusses its applications in statistical learning, deep learning and reinforcement learning.Then the paper presents the studies of typical UEBA algorithms, and gives comparative analysis of their performance.The paper also describes several commonly used public datasets, feature engineering methods, and two feature processing methods that enhance the accuracy of behavior representation.On this basis, this paper summarizes the advantages and disadvantages of typical anomaly detection algorithms, and the limitations of internal threat analysis and external intrusion detection.Finally, the future research directions in this field are discussed.

Key words: network security, User and Entity Behavior Analytics(UEBA), anomaly detection, statistical learning, deep learning, reinforcement learning

中图分类号:

TP393

崔景洋, 陈振国, 田立勤, 张光华. 基于机器学习的用户与实体行为分析技术综述[J]. 计算机工程, 2022, 48(2): 10-24.

CUI Jingyang, CHEN Zhenguo, TIAN Liqin, ZHANG Guanghua. Overview of User and Entity Behavior Analytics Technology Based on Machine Learning[J]. Computer Engineering, 2022, 48(2): 10-24.

https://www.ecice06.com/CN/Y2022/V48/I2/10

图/表 12

20220226123956

20220226124000

20220226124004

20220226124007

20220226124011

20220226124015

20220226124020

20220226124024

20220226124031

20220226124034

20220226124038

20220226124042

参考文献

[1] CHANG L, CHEN W, HUANG J B, et al.Exploiting multi-attention network with contextual influence for point-of-interest recommendation[J].Applied Intelligence, 2021, 51(4):1904-1917.
[2] ALHIJAWI B, AL-NAYMAT G, OBEID N, et al.Novel predictive model to improve the accuracy of collaborative filtering recommender systems[J].Information Systems, 2021, 96:1-30.
[3] ZHU X, LIU H, LEI Z, et al.Large-scale bisample learning on ID vs.spot face recognition[J].International Journal of Computer Vision, 2019, 127(6):684-700.
[4] MURALIDHARAN A, MOSTOFI Y.Path planning for minimizing the expected cost until success[J].IEEE Transactions on Robotics, 2019, 35(2):466-481.
[5] JAGIELSKI M, OPREA A, BIGGIO B, et al.Manipulating machine learning:poisoning attacks and countermeasures for regression learning[C]//Proceedings of 2018 IEEE Symposium on Security and Privacy.Washington D.C., USA:IEEE Press, 2018:19-35.
[6] BÉCUE A, PRAÇA I, GAMA J.Artificial intelligence, cyber-threats and industry 4.0:challenges and opportunities[J].Artificial Intelligence Review, 2021, 54(5):3849-3886.
[7] KAUR H, PANNU H S, MALHI A K.A systematic review on imbalanced data challenges in machine learning[J].ACM Computing Surveys, 2019, 52(4):1-36.
[8] SINGH K, SINGH P, KUMAR K.User behavior analytics-based classification of application layer HTTP-GET flood attacks[J].Journal of Network and Computer Applications, 2018, 112:97-114.
[9] SHASHANKA M, SHEN M Y, WANG J S.User and entity behavior analytics for enterprise security[C]//Proceedings of 2016 IEEE International Conference on Big Data.Washington D.C., USA:IEEE Press, 2016:1867-1874.
[10] ALEXEY L, MIKHAIL P, ANATOLIY B.Scalable data processing approach and anomaly detection method for user and entity behavior analytics platform[C]//Proceedings of 2020 International Symposium on Intelligent and Distributed Computing.Berlin, Germany:Springer, 2020:344-349.
[11] GUPTA R, TANWAR S, TYAGI S, et al.Machine learning models for secure data analytics:a taxonomy and threat model[J].Computer Communications, 2020, 153:406-440.
[12] 文雨, 王伟平, 孟丹.面向内部威胁检测的用户跨域行为模式挖掘[J].计算机学报, 2016, 39(8):1555-1569. WEN Y, WANG W P, MENG D.Mining user cross-domain behavior patterns for insider threat detection[J].Chinese Journal of Computers, 2016, 39(8):1555-1569.(in Chinese)
[13] 李志, 宋礼鹏.基于用户窗口行为的内部威胁检测研究[J].计算机工程, 2020, 46(4):135-142, 150. LI Z, SONG L P.Research on internal threat detection based on user window behavior[J].Computer Engineering, 2020, 46(4):135-142, 150.(in Chinese)
[14] YANG A M, LIU C S, LI J, et al.Design of intrusion detection system for Internet of Things based on improved BP neural network[J].IEEE Access, 2019, 7:106043-106052.
[15] AHMIM A, DERDOUR M, FERRAG M A.An intrusion detection system based on combining probability predictions of a tree of classifiers[J].International Journal of Communication Systems, 2018, 31(9):1-17.
[16] BELOUCH M, EL S, IDHAMMAD M.A two-stage classifier approach using RepTree algorithm for network intrusion detection[J].International Journal of Advanced Computer Science and Applications, 2017, 8(6):389-394.
[17] SINGH S, YASSINE A.Big data mining of energy time series for behavioral analytics and energy consumption forecasting[J].Energies, 2018, 11(2):452.
[18] YU H, ZHANG T, CHEN J, et al.Web items recommendation based on multi-view clustering[C]//Proceedings of the 42nd IEEE Computer Software & Applications Conference.Washington D.C., USA:IEEE Press, 2018:420-425.
[19] Gartner.Market guide for user and entity behavior analytics[EB/OL].[2021-08-12].https://www.gartner.com/en/documents/3134524.
[20] LUNT T F.A survey of intrusion detection techniques[J].Computers & Security, 1993, 12(4):405-418.
[21] LUNT T F, JAGANNATHAN R.A prototype real-time intrusion-detection expert system[C]//Proceedings of 1988 IEEE Symposium on Security and Privacy.Washington D.C., USA:IEEE Press, 1988:59-66.
[22] HOGLUND G W, VALCARCE E M.The "ESSENSE" of intrusion detection:a knowledge-based approach to security monitoring and control[C]//Proceedings of the 7th International Conference on Industrial and Engineering Applications of Artificial Intelligence and Expert Systems.New York, USA:ACM Press, 1994:201-209.
[23] KHAN M A, ABUHASEL K A.An evolutionary multi-hidden Markov model for intelligent threat sensing in industrial Internet of Things[J].The Journal of Supercomputing, 2021, 77(6):6236-6250.
[24] ZHANG G, YIN J, LIANG Z H, et al.Prior knowledge SVM-based intrusion detection framework[C]//Proceedings of the 3rd International Conference on Natural Computation.Washington D.C., USA:IEEE Press, 2007:489-493.
[25] LEWICKI A, PANCERZ K.Ant-based clustering for flow graph mining[J].International Journal of Applied Mathematics and Computer Science, 2020, 30(2):561-572.
[26] MIAH S J, VU H Q, GAMMACK J, et al.A big data analytics method for tourist behaviour analysis[J].Information & Management, 2017, 54(6):771-785.
[27] WANG K, ZHENG H, LOURI A.TSA-NoC:learning-based threat detection and mitigation for secure network-on-chip architecture[J].IEEE Micro, 2020, 40(5):56-63.
[28] XU S J, QIAN Y, HU R Q.Edge intelligence assisted gateway defense in cyber security[J].IEEE Network, 2020, 34(4):14-19.
[29] RAHUL-VIGNESWARAN K, POORNACHANDRAN P, SOMAN K P.A compendium on network and host based intrusion detection systems[C]//Proceedings of International Conference on Data Science, Machine Learning and Applications.Berlin, Germany:Springer, 2019:23-30.
[30] 李航.统计学习方法[M].北京:清华大学出版社, 2012. LI H.Statistical learning method[M].Beijing:Tsinghua University Press, 2012.(in Chinese)
[31] SUABOOT J, FAHAD A, TARI Z, et al.A taxonomy of supervised learning for IDSs in SCADA environments[J].ACM Computing Surveys, 2020, 53(2):1-37.
[32] PICCIALLI F, CASOLLA G, CUOMO S, et al.Decision making in IoT environment through unsupervised learning[J].IEEE Intelligent Systems, 2020, 35(1):27-35.
[33] VILLA-PÉREZ M E, ÁLVAREZ-CARMONA M Á, LOYOLA-GONZÁLEZ O, et al.Semi-supervised anomaly detection algorithms:a comparative summary and future research directions[J].Knowledge-Based Systems, 2021, 218:1-18.
[34] GARCÍA S, LUENGO J, SÁEZ J A, et al.A survey of discretization techniques:taxonomy and empirical analysis in supervised learning[J].IEEE Transactions on Knowledge and Data Engineering, 2013, 25(4):734-750.
[35] KEMALIS K, TZOURAMANIS T.SQL-IDS:a specification-based approach for SQL-injection detection[C]//Proceedings of 2008 ACM Symposium on Applied Computing.New York, USA:ACM Press, 2008:2153-2158.
[36] GU Y P, YU X, GUO K X, et al.Detection, estimation, and compensation of false data injection attack for UAVs[J].Information Sciences, 2021, 546:723-741.
[37] BRONTE R, SHAHRIAR H, HADDAD H M.A signature-based intrusion detection system for Web applications based on genetic algorithm[C]//Proceedings of the 9th International Conference on Security of Information and Networks.New York, USA:ACM Press, 2016:32-39.
[38] GARCÍA-TEODORO P, DÍAZ-VERDEJO J.Anomaly-based network intrusion detection:techniques, systems and challenges[J].Computers & Security, 2009, 28(1/2):18-28.
[39] HUTCHINS R, ZEGURA E W, LIASHENKO A, et al.Internet user access via dial-up networks-traffic characterization and statistics[C]//Proceedings the 9th International Conference on Network Protocols.Washington D.C., USA:IEEE Press, 2001:314-322.
[40] KIM M.Network traffic prediction based on INGARCH model[J].Wireless Networks, 2020, 26(8):6189-6202.
[41] DI GESU V, LO BOSCO G, FRIEDMAN J H.Intruders pattern identification[C]//Proceedings of the 19th International Conference on Pattern Recognition.Washington D.C., USA:IEEE Press, 2008:1-4.
[42] SANDOSH S, GOVINDASAMY V, AKILA G.Enhanced intrusion detection system via Agent clustering and classification based on outlier detection[J].Peer-to-Peer Networking and Applications, 2020, 13(3):1038-1045.
[43] LU C W, SHI J P, WANG W M, et al.Fast abnormal event detection[J].International Journal of Computer Vision, 2019, 127(8):993-1011.
[44] SU C, CAO J.Improving lazy decision tree for imbalanced classification by using skew-insensitive criteria[J].Applied Intelligence, 2019, 49(3):1127-1145.
[45] SOYSAL M, SCHMIDT E G.Machine learning algorithms for accurate flow-based network traffic classification:evaluation and comparison[J].Performance Evaluation, 2010, 67(6):451-467.
[46] AL-YASEEN W L, OTHMAN Z A, NAZRI M Z A.Multi-level hybrid support vector machine and extreme learning machine based on modified K-Means for intrusion detection system[J].Expert Systems with Applications, 2017, 67:296-303.
[47] 何发镁, 马慧珍, 王旭仁, 等.基于特征分组聚类的异常入侵检测系统研究[J].计算机工程, 2020, 46(4):123-128, 134. HE F M, MA H Z, WANG X R, et al.Research on anomaly intrusion detection system based on feature grouping clustering[J].Computer Engineering, 2020, 46(4):123-128, 134.(in Chinese)
[48] CHEN Y W, TANG S Y, BOUGUILA N, et al.A fast clustering algorithm based on pruning unnecessary distance computations in DBSCAN for high-dimensional data[J].Pattern Recognition, 2018, 83:375-387.
[49] TANG D, ZHANG S Q, CHEN J W, et al.The detection of low-rate DoS attacks using the SADBSCAN algorithm[J].Information Sciences, 2021, 565:229-247.
[50] BIAN Z K, CHUNG F L, WANG S T.Fuzzy density peaks clustering[J].IEEE Transactions on Fuzzy Systems, 2021, 29(7):1725-1738.
[51] RONAO C A, CHO S B.Mining SQL queries to detect anomalous database access using random forest and PCA[C]//Proceedings of International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems.Berlin, Germany:Springer, 2015:151-160.
[52] 王晓东, 赵一宁, 肖海力, 等.多节点系统异常日志流量模式检测方法[J].软件学报, 2020, 31(10):3295-3308. WANG X D, ZHAO Y N, XIAO H L, et al.Multi-node system abnormal log flow mode detection method[J].Journal of Software, 2020, 31(10):3295-3308.(in Chinese)
[53] ZHU X J, GOLDBERG A B.Introduction to semi-supervised learning[J].Synthesis Lectures on Artificial Intelligence and Machine Learning, 2009, 3(1):1-6.
[54] 李杰铃, 张浩.半监督异常流量检测研究综述[J].小型微型计算机系统, 2020, 41(11):2371-2379. LI J L, ZHANG H.Survey on semi-supervised anomaly traffic detection[J].Journal of Chinese Computer Systems, 2020, 41(11):2371-2379.(in Chinese)
[55] TAHA A, HADI A S.Anomaly detection methods for categorical data[J].ACM Computing Surveys, 2019, 52(2):1-35.
[56] IDHAMMAD M, AFDEL K, BELOUCH M.Semi-supervised machine learning approach for DDoS detection[J].Applied Intelligence, 2018, 48(10):3193-3208.
[57] GU Y H, LI K Y, GUO Z Y, et al.Semi-supervised K-means DDoS detection method using hybrid feature selection algorithm[J].IEEE Access, 2019, 7:64351-64365.
[58] LIU Y, XU Z, LI C G.Online semi-supervised support vector machine[J].Information Sciences, 2018, 439:125-141.
[59] GUO H J, ZOU H, TAN J Y.Semi-supervised dimensionality reduction via sparse locality preserving projection[J].Applied Intelligence, 2020, 50(4):1222-1232.
[60] DE VRIES S, THIERENS D.A reliable ensemble based approach to semi-supervised learning[J].Knowledge-Based Systems, 2021, 215:106-121.
[61] SUN J Y, WANG X Z, XIONG N X, et al.Learning sparse representation with variational auto-encoder for anomaly detection[J].IEEE Access, 2018, 6:33353-33361.
[62] ZHOU C, PAFFENROTH R C.Anomaly detection with robust deep autoencoders[C]//Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York, USA:ACM Press, 2017:23-40.
[63] CAO V L, NICOLAU M, MCDERMOTT J.A hybrid autoencoder and density estimation model for anomaly detection[C]//Proceedings of 2016 International Conference on Parallel Problem Solving from Nature.Berlin, Germany:Springer, 2016:717-726.
[64] ROY S S, MALLIK A, GULATI R, et al.A deep learning based artificial neural network approach for intrusion detection[C]//Proceedings of 2017 International Conference on Mathematics and Computing.Berlin, Germany:Springer, 2017:44-53.
[65] SAFARA F, SOURI A, SERRIZADEH M.Improved intrusion detection method for communication networks using association rule mining and artificial neural networks[J].IET Communications, 2020, 14(7):1192-1197.
[66] WANG W, ZHU M, ZENG X W, et al.Malware traffic classification using convolutional neural network for representation learning[C]//Proceedings of 2017 International Conference on Information Networking.Washington D.C., USA:IEEE Press, 2017:712-717.
[67] PRIYANGA S P, KRITHIVASAN K, S P, et al.Detection of cyberattacks in industrial control systems using Enhanced Principal Component Analysis and Hypergraph-based Convolution Neural Network(EPCA-HG-CNN)[J].IEEE Transactions on Industry Applications, 2020, 56(4):4394-4404.
[68] HU Y B, ZHANG D H, CAO G Y, et al.Network data analysis and anomaly detection using CNN technique for industrial control systems security[C]//Proceedings of 2019 IEEE International Conference on Systems, Man and Cybernetics.Washington D.C., USA:IEEE Press, 2019:593-597.
[69] 李峰, 舒斐, 李明轩, 等.基于深度学习的Linux远控木马检测[J].计算机工程, 2020, 46(7):159-164. LI F, SHU F, LI M X, et al.Detection of remote access trojan in Linux based on deep learning[J].Computer Engineering, 2020, 46(7):159-164.(in Chinese)
[70] 徐洪平, 马泽文, 易航, 等.基于卷积循环神经网络的网络流量异常检测技术[J].信息网络安全, 2021, 21(7):54-62. XU H P, MA Z W, YI H, et al.Network traffic anomaly detection technology based on convolutional recurrent neural network[J].Netinfo Security, 2021, 21(7):54-62. (in Chinese)
[71] LIU H Y, LANG B, LIU M, et al.CNN and RNN based payload classification methods for attack detection[J].Knowledge-Based Systems, 2019, 163:332-341.
[72] MANICKAM M, RAMARAJ N, CHELLAPPAN C.A combined PFCM and recurrent neural network-based intrusion detection system for cloud environment[J].International Journal of Business Intelligence and Data Mining, 2019, 14(4):504-527.
[73] 麻文刚, 张亚东, 郭进.基于LSTM与改进残差网络优化的异常流量检测方法[J].通信学报, 2021, 42(5):23-40. MA W G, ZHANG Y D, GUO J.Abnormal traffic detection method based on LSTM and improved residual neural network optimization[J].Journal on Communications, 2021, 42(5):23-40.(in Chinese)
[74] GOODFELLOW I, POUGET-ABADIE J, MIRZA M, et al.Generative adversarial networks[J].Communications of the ACM, 2020, 63(11):139-144.
[75] ZHUO Y, GE Z Q.Gaussian discriminative analysis aided GAN for imbalanced big data augmentation and fault classification[J].Journal of Process Control, 2020, 92:271-287.
[76] USAMA M, ASIM M, LATIF S, et al.Generative adversarial networks for launching and thwarting adversarial attacks on network intrusion detection systems[C]//Proceedings of the 15th International Wireless Communications & Mobile Computing Conference.Washington D.C., USA:IEEE Press, 2019:78-83.
[77] 席亮, 刘涵, 樊好义, 等.基于深度对抗学习潜在表示分布的异常检测模型[J].电子学报, 2021, 49(7):1257-1265. XI L, LIU H, FAN H Y, et al.Deep adversarial learning latent representation distribution model for anomaly detection[J].Acta Electronica Sinica, 2021, 49(7):1257-1265.(in Chinese)
[78] ZHANG X Q, ZHOU Y, PEI S W, et al.Adversarial examples detection for XSS attacks based on generative adversarial networks[J].IEEE Access, 2020, 8:10989-10996.
[79] ZHANG X.Network intrusion detection using generative adversarial networks[D].Christchurch, New Zealand:University of Canterbury, 2020.
[80] BARTO A G.Reinforcement learning:connections, surprises, and challenge[J].AI Magazine, 2019, 40(1):3-15.
[81] NGUYEN T T, NGUYEN N D, NAHAVANDI S.Deep reinforcement learning for multiagent systems:a review of challenges, solutions, and applications[J].IEEE Transactions on Cybernetics, 2020, 50(9):3826-3839.
[82] DEMONTIS A, MELIS M, BIGGIO B, et al.Yes, machine learning can be more secure! A case study on android malware detection[J].IEEE Transactions on Dependable and Secure Computing, 2019, 16(4):711-724.
[83] 高洋, 王礼伟, 任望, 等.基于强化学习的工控系统恶意软件行为检测方法[J].工程科学学报, 2020, 42(4):455-462. GAO Y, WANG L W, REN W, et al.Reinforcement learning-based detection method for malware behavior in industrial control systems[J].Chinese Journal of Engineering, 2020, 42(4):455-462.(in Chinese)
[84] XIAO L, XU D J, MANDAYAM N B, et al.Attacker-centric view of a detection game against advanced persistent threats[J].IEEE Transactions on Mobile Computing, 2018, 17(11):2512-2523.
[85] XIAO L, DING Y Z, JIANG D H, et al.A reinforcement learning and blockchain-based trust mechanism for edge networks[J].IEEE Transactions on Communications, 2020, 68(9):5460-5470.
[86] CHEN Y, HUANG S W, LIU F, et al.Evaluation of reinforcement learning-based false data injection attack to automatic voltage control[J].IEEE Transactions on Smart Grid, 2019, 10(2):2158-2169.
[87] University of California.KDD CUP 1999 data[EB/OL].[2021-08-12].http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.
[88] MIT Lincoln Laboratory.1998 DARPA intrusion detection evaluation dataset[EB/OL].[2021-08-12].http://www.ll.mit.edu/r-d/datasets/1998-darpa-intrusion-detection-evaluation-dataset.
[89] MIT Lincoln Laboratory.1999 DARPA intrusion detection evaluation dataset[EB/OL].[2021-08-12].http://www.ll.mit.edu/r-d/datasets/1999-darpa-intrusion-detection-evaluation-dataset.
[90] MIT Lincoln Laboratory.2000 DARPA intrusion detection scenario specific datasets[EB/OL].[2021-08-12].http://www.ll.mit.edu/r-d/datasets/2000-darpa-intrusion-detection-scenario-specific-datasets.
[91] Canadian Institute for Cybersecurity.NSL-KDD datasets[EB/OL].[2021-08-12].https://www.unb.ca/cic/datasets/nsl.html.
[92] University of New South Wales.The UNSW-NB15 dataset[EB/OL].[2021-08-12].https://research.unsw.edu.au/projects/unsw-nb15-dataset.
[93] Canadian Institute for Cybersecurity.Intrusion detection evaluation dataset(CIC-IDS2017)[EB/OL].[2021-08-12].https://www.unb.ca/cic/datasets/ids-2017.html.
[94] Canadian Institute for Cybersecurity.IDS 2012 datasets[EB/OL].[2021-08-12].https://www.unb.ca/cic/datasets/ids.html.
[95] Canadian Institute for Cybersecurity.IDS 2018 datasets[EB/OL].[2021-08-12].https://www.unb.ca/cic/datasets/ids-2018.html.
[96] Canadian Institute for Cybersecurity.Datasets research[EB/OL].[2021-08-12].https://www.unb.ca/cic/datasets/index.html.
[97] DUMOUCHEL W, JU W H, KARR A F, et al.Computer intrusion:detecting masquerades[J].Statistical Science, 2001, 16(1):58-74.
[98] LAMPSON B W.Computer security in the real world[J].Computer, 2004, 37(6):37-46.

选择文件类型/文献管理软件名称

选择包含的内容