Detection of Remote Access Trojan in Linux Based on Deep Learning

doi:10.19678/j.issn.1000-3428.0054943

Abstract

Abstract: As a high-level form of malicious code,Remote Access Trojan(RAT) can be used to collect sensitive user information and even launch large-scale attacks through command control.To accurately detect RAT,this paper proposes a new deep learning-based method that combines static analysis with dynamic behavior analysis to extract file features.By taking advantage of the ability of deep learning to extract sample features layer by layer,this method constructs a sample classification model based on Recurrent Neural Network(RNN) to detect RAT in Linux.Further,in order to avoid being trapped in local optima,random search of parameters is adopted to select hyperparameter of the model.Experimental results show that compared with other models based on traditional machine learning algorithms,the proposed RNN-based sample classification model has higher accuracy and F1 value when selecting the hyperparameter configuration with the best performance.

Key words: Remote Access Trojan(RAT), static analysis, behavior analysis, Recurrent Neural Network(RNN), hyperparameter

摘要： 远控木马作为一种高级形态的恶意代码，不仅能收集用户敏感信息，而且可以通过命令控制引发大规模的攻击。为高效准确地识别远控木马，通过结合静态分析和动态行为分析方法提取文件特征，利用深度学习对样本特征逐层抽取的能力，构建基于循环神经网络（RNN）的样本分类模型，以对Linux远控木马进行检测。为避免陷入局部最优，采用随机搜索参数的方法进行模型超参数选择。对基于RNN的分类模型及其他基于传统机器学习算法的模型分别进行实验，结果表明，在选取性能最佳的超参数配置下，基于RNN的样本分类模型具有更高的准确率与F1值。

关键词: 远控木马, 静态分析, 行为分析, 循环神经网络, 超参数

CLC Number:

TP391

LI Feng, SHU Fei, LI Mingxuan, WANG Bin, YANG Huiting. Detection of Remote Access Trojan in Linux Based on Deep Learning[J]. Computer Engineering, 2020, 46(7): 159-164.

李峰, 舒斐, 李明轩, 王斌, 杨慧婷. 基于深度学习的Linux远控木马检测[J]. 计算机工程, 2020, 46(7): 159-164.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0054943

http://www.ecice06.com/EN/Y2020/V46/I7/159

References

[1] MAHENDRAN A,VEDALDI A.Understanding deep image representations by inverting them[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition.Washington D.C.,USA:IEEE Press,2015:5188-5196.
[2] DENG L.Deep learning:methods and applications[J].Foundations and Trends in Signal Processing,2014,7(3/4):197-387.
[3] VISHWANATHAN S V M,MURTY M N.SSVM:a simple SVM algorithm[C]//Proceedings of 2002 International Joint Conference on Neural Networks.Washington D.C.,USA:IEEE Press,2002:2393-2398.
[4] ZHANG M L,ZHOU Z H.ML-KNN:a lazy learning approach to multi-label learning[J].Pattern Recognition,2007,40(7):2038-2048.
[5] SAFAVIAN S R,LANDGREBE D.A survey of decision tree classifier methodology[J].IEEE Transactions on Systems,Man,and Cybernetics,1991,21(3):660-674.
[6] LIAW A,WIENER M.Classification and regression by random forest[J].R News,2002,2(3):18-22.
[7] SAXE J,BERLIN K.Deep neural network based malware detection using two dimensional binary program features[C]//Proceedings of 2015 International Conference on Malicious and Unwanted Software.Washington D.C.,USA:IEEE Press,2015:11-20.
[8] GROSSE K,PAPERNOT N,MANOHARAN P,et al.Adversarial examples for malware detection[C]//Proceedings of European Symposium on Research in Computer Security.Berlin,Germany:Springer,2017:62-79.
[9] TOBIYAMA S,YAMAGUCHI Y,SHIMADA H,et al.Malware detection with deep neural network using process behavior[C]//Proceedings of 2016 IEEE Annual Computer Software and Applications Conference.Washington D.C.,USA:IEEE Press,2016:577-582.
[10] TIAN R,ISLAM R,BATTEN L,et al.Differentiating malware from cleanware using behavioural analysis[C]//Proceedings of 2010 International Conference on Malicious and Unwanted Software.Washington D.C.,USA:IEEE Press,2010:23-30.
[11] COZZI E,GRAZIANO M,FRATANTONIO Y,et al.Understanding Linux malware[EB/OL].[2019-04-20].http://www.s3.eurecom.fr/docs/oakland18_cozzi.pdf.
[12] MARCOS S,RIVERA R,KOTZIAS P,et al.AVclass:a tool for massive malware labeling[C]//Proceedings of International Symposium on Research in Attacks,Intrusions,and Defenses.Berlin,Germany:Springer,2016:263-276.
[13] WANG Xiajing,HU Changzhen,MA Rui,et al.A survey of the key technology of binary program vulnerability discovery[J].Netinfo Security,2017(8):1-13.(in Chinese)王夏菁,胡昌振,马锐,等.二进制程序漏洞挖掘关键技术研究综述[J].信息网络安全,2017(8):1-13.
[14] Systemtap[EB/OL].[2019-04-20].https://sourceware.org/systemtap.
[15] WU Di,FANG Binxing,CUI Xiang,et al.BotCatcher:botnet detection system based on deep learning[J].Journal on Communications,2018,39(8):18-28.(in Chinese)吴迪,方滨兴,崔翔,等.BotCatcher:基于深度学习的僵尸网络检测系统[J].通信学报,2018,39(8):18-28.
[16] WANG Guihuai,ZHONG Cheng,CHU Xiumin,et al.Ship trajectory restoration method based on BLSTM-RNN[J].Journal of Chongqing Jiaotong University(Natural Sciences),2019,38(10):7-12,67.(in Chinese)王贵槐,钟诚,初秀民,等.基于BLSTM-RNN的船舶轨迹修复方法[J].重庆交通大学学报(自然科学版),2019,38(10):7-12,67.
[17] RUI F,ZUO Z,LI L.Using LSTM and GRU neural network methods for traffic flow prediction[C]//Proceedings of Youth Academic Conference of Chinese Association of Automation.Washington D.C.,USA:IEEE Press,2017:266-276.
[18] RHODE M,BURNAP P,JONES K.Early-stage malware prediction using recurrent neural networks[J].Computers & Security,2018,77:578-594.
[19] NATHANLOPE Z.Python remote administration tool[EB/OL].[2019-04-20].https://github.com/nathanlopez/Stitch.
[20] HUANG Wenming,MO Yang.Chinese spam message filtering based on text weighted KNN algorithm[J].Computer Engineering,2017,43(3):193-199.(in Chinese)黄文明,莫阳.基于文本加权KNN算法的中文垃圾短信过滤[J].计算机工程,2017,43(3):193-199.
[21] FANG Yong,ZHU Guangxiatian,LIU Luping,et al.Research on browser Fuzz sample generation technology based on deep learning[J].Netinfo Security,2019,19(3):26-33.(in Chinese)方勇,朱光夏天,刘露平,等.基于深度学习的浏览器Fuzz样本生成技术研究[J].信息网络安全,2019,19(3):26-33.

Please choose a citation manager

Content to export