Kernel Rootkit Detection Method Based on Multidimensional View Tracking

doi:10.19678/j.issn.1000-3428.0069075

Abstract

Abstract:

In Linux servers, kernel Rootkit can be concealed in the operating system for a long time, causing serious kernel damage. In particular, unknown Rootkit, with random attack occurrence time and spatial distribution, pose a significant challenge to discovering the source of an attack. Because the source code is unknown, conventional methods face difficulties in analyzing its behavioral characteristics and are unable to pre-set detection points at appropriate locations. To address this threat, this study proposes a kernel Rootkit detection method based on multidimensional view tracing. By cross-comparing multiple views in both the spatial and temporal dimensions, the malicious behavior of unknown kernel Rootkit is detected and hidden data are restored. Experiments and analyses show that the proposed method is efficient in detecting kernel Rootkit, with a CPU overhead of only 0.38% in the case of a secure response cycle of 0.1 s.

Key words: kernel Rootkit, Linux server security, multidimensional view tracking, hidden detection, restore hidden data

摘要：

在Linux服务器中, 内核Rootkit因其高隐蔽性和高特权性, 可长时间潜伏在操作系统中, 对内核造成严重破坏, 尤其是未知Rootkit, 其攻击发生时间和空间分布都是随机的, 这为发现攻击源头带来严峻挑战。由于源码是未知的, 常规方法难以分析其行为特征, 也无法在合适的位置预设检测点。为了应对这一威胁, 提出一种基于多维视图追踪的内核Rootkit检测方法。通过在空间维度和时间维度上对多个视图进行交叉对比, 可检测未知内核Rootkit的恶意行为, 并还原被隐藏的数据。实验和分析显示, 该方法对内核Rootkit具有良好的检测效果, 在安全响应周期为0.1 s的情况下, 该方法引入的CPU开销仅为0.38%。

关键词: 内核Rootkit, Linux服务器安全, 多维视图追踪, 隐藏检测, 还原隐藏数据

ZHENG Yijian, LI Yonggang. Kernel Rootkit Detection Method Based on Multidimensional View Tracking[J]. Computer Engineering, 2026, 52(3): 177-189.

郑伊健, 李勇钢. 基于多维视图追踪的内核Rootkit检测方法[J]. 计算机工程, 2026, 52(3): 177-189.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069075

https://www.ecice06.com/EN/Y2026/V52/I3/177

Figures/Tables 17

Fig.1 The overall architecture of the system in this paper

Fig.2 Module detection method

Fig.3 Process detection method

Fig.4 Construction method of process scanning view

Fig.5 The mapping relationship between task_struct and socket

Fig.6 Port detection method

Fig.7 The process of locating the root directory

Fig.8 PTE corresponding to executable code segments of kernel modules

Fig.9 The process of dynamically tracking control flow

Fig.10 Kernel module attack chain based on exported functions

Fig.11 Detection results of kernel module attack chain

Fig.12 The runtime CPU overhead introduced under different cycles

Fig.13 System latency

Fig.14 I/O overhead

References 49

1	NADIM M, AKOPIAN D, LEE W. A review on learning-based detection approaches of the kernel-level Rootkit[C]//Proceedings of the International Conference on Engineering and Emerging Technologies (ICEET). Washington D.C., USA: IEEE Press, 2022: 1-6.
2	JOY J, JOHN A, JOY J. Rootkit detection mechanism: a survey[EB/OL]. [2023-07-05]. https://www.researchgate.net/publication/216463350_Rootkit_Detection_Mechanism_A_Survey.
3	CATUOGNO L, GALDI C. Ensuring application integrity: a survey on techniques and tools[C]//Proceedings of the 9th International Conference on Innovative Mobile and Internet Services in Ubiquitous Computing. Washington D.C., USA: IEEE Press, 2015: 192-199.
4	MOON D , PAN S B , KIM I . Host-based intrusion detection system for secure human-centric computing. The Journal of Supercomputing, 2016, 72(7): 2520- 2536. doi: 10.1007/s11227-015-1506-9
5	OHELLO S P, SUHARD I. Android malware evasion framework for auditing anti-malware resistance against various obfuscation technique and dynamic code loading[C]//Proceedings of the International Conference on Information Technology Systems and Innovation (ICITSI). Washington D.C., USA: IEEE Press, 2022: 183-188.
6	SYED N F , SHAH S W , TRUJILLO-RASUA R , et al. Traceability in supply chains: a cyber security analysis. Computers & Security, 2022, 112, 102536.
7	LI Y G , CHUNG Y C , HWANG K , et al. Virtual wall: filtering Rootkit attacks to protect Linux kernel functions. IEEE Transactions on Computers, 2021, 70(10): 1640- 1653. doi: 10.1109/TC.2020.3022023
8	NADIM M, LEE W, AKOPIAN D. Kernel-level Rootkit detection, prevention and behavior profiling: a taxonomy and survey[EB/OL]. [2023-07-05]. https://arxiv.org/pdf/2304.00473.
9	MAHAPATRA C , SELVAKUMAR S . An online cross view difference and behavior based kernel Rootkit detector. ACM SIGSOFT Software Engineering Notes, 2011, 36(4): 1- 9.
10	LIANG B, YOU W, SHI W C, et al. Detecting stealthy malware with inter-structure and imported signatures[C]//Proceedings of the 6th ACM Symposium on Information, Computer and Communications Security. New York, USA: ACM Press, 2011: 217-227.
11	WANG Y M, BECK D, VO B, et al. Detecting stealth software with Strider GhostBuster[C]//Proceedings of the International Conference on Dependable Systems and Networks (DSN'05). Washington D.C., USA: IEEE Press, 2005: 368-377.
12	KORKIN I, NESTEROV I. Applying memory forensics to Rootkit detection[EB/OL]. [2023-07-05]. https://arxiv.org/abs/1506.04129.
13	RHEE J, RILEY R, XU D Y, et al. Kernel malware analysis with Un-tampered and temporal views of dynamic kernel memory[EB/OL]. [2023-07-05]. https://link.springer.com/chapter/10.1007/978-3-642-15512-3_10.
14	SUDALAIMUTHU T. Kernel Rootkit secret detection in cloud computing[C]//Proceedings of the 1st International Conference on Computational Science and Technology (ICCST). Washington D.C., USA: IEEE Press, 2023: 276-280.
15	WAMPLER D, GRAHAM J. A method for detecting Linux kernel module Rootkits[EB/OL]. [2023-07-05]. https://link.springer.com/chapter/10.1007/978-0-387-73742-3_7.
16	WAMPLER D , GRAHAM J H . A normality based method for detecting kernel Rootkits. ACM SIGOPS Operating Systems Review, 2008, 42(3): 59- 64. doi: 10.1145/1368506.1368515
17	XU L, SU Z. Dynamic detection of process-hiding kernel Rootkits[EB/OL]. [2023-07-05]. https://www.semanticscholar.org/paper/Dynamic-Detection-of-Process-Hiding-Kernel-Rootkits-Xu-Su/a8b01e12a713d64d8c92665375995750149386ff.
18	WANG Y, HU C M, LI B. VMDetector: a VMM-based platform to detect hidden process by multi-view comparison[C]//Proceedings of the 13th IEEE International Symposium on High-Assurance Systems Engineering. Washington D.C., USA: IEEE Press, 2011: 307-312.
19	XIE X W, WANG W C. Rootkit detection on virtual machines through deep information extraction at hypervisor-level[C]//Proceedings of the IEEE Conference on Communications and Network Security (CNS). Washington D.C., USA: IEEE Press, 2013: 498-503.
20	VÖMEL S, LENZ H. Visualizing indicators of Rootkit infections in memory forensics[C]//Proceedings of the 7th International Conference on IT Security Incident Management and IT Forensics. Washington D.C., USA: IEEE Press, 2013: 122-139.
21	XIAO J D, LU L, WANG H N, et al. HyperLink: virtual machine introspection and memory forensic analysis without kernel source code[C]//Proceedings of the IEEE International Conference on Autonomic Computing (ICAC). Washington D.C., USA: IEEE Press, 2016: 127-136.
22	HUA Q, ZHANG Y. Detecting malware and Rootkit via memory forensics[C]//Proceedings of the International Conference on Computer Science and Mechanical Automation (CSMA). Washington D.C., USA: IEEE Press, 2016: 92-96.
23	DI PIETRO R, FRANZONI F, LOMBARDI F. HyBIS: advanced introspection for effective windows guest protection[EB/OL]. [2023-07-05]. https://link.springer.com/chapter/10.1007/978-3-319-58469-0_13.
24	LAMPS J, PALMER I, SPRABERY R. WinWizard: expanding Xen with a LibVMI intrusion detection tool[C]//Proceedings of the 7th IEEE International Conference on Cloud Computing. Washington D.C., USA: IEEE Press, 2014: 849-856.
25	ZAKI A , HUMPHREY B . Unveiling the kernel: Rootkit discovery using selective automated kernel memory differencing. Virus Bulletin, 2014, 12, 239- 256.
26	CUI W, PEINADO M, XU Z, et al. Tracking Rootkit footprints with a practical memory analysis system[EB/OL]. [2023-07-05]. https://www.usenix.org/system/files/conference/usenixsecurity12/sec12-final196.pdf.
27	CASE A , RICHARD G G . Advancing Mac OS X Rootkit detection. Digital Investigation, 2015, 14, S25- S33. doi: 10.1016/j.diin.2015.05.005
28	NAGY R, BUTTYÁN L. Rootkit detection with memory snapshot inspection on embedded IoT devices[EB/OL]. [2023-07-05]. https://static.crysys.hu/publications/files/setit/thesis_bme_Nagy21msc.pdf.
29	ZHOU H W , SHI W C , YUAN J H , et al. BeCFI: detecting hidden control flow with performance monitoring counters. International Journal of High Performance Computing and Networking, 2016, 9(5/6): 470- 479. doi: 10.1504/IJHPCN.2016.080420
30	LI Y G , WU Y , CUI C Y , et al. RMVP: a real-time method to monitor random processes of virtual machine. IEEE Access, 2019, 7, 15845- 15860. doi: 10.1109/ACCESS.2019.2893627
31	QUYNH N A, TAKEFUJI Y. Towards a tamper-resistant kernel Rootkit detector[C]//Proceedings of the 2007 ACM Symposium on Applied Computing. New York, USA: ACM Press, 2007: 276-283.
32	CONNELLY J, ROBERTS T, GAO X, et al. CloudSkulk: a nested virtual machine based Rootkit and its detection[C]//Proceedings of the 51st Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). Washington D.C., USA: IEEE Press, 2021: 350-362.
33	MANōS V J M , JANG D , RYU C , et al. Domain isolated kernel: a lightweight sandbox for untrusted kernel extensions. Computers & Security, 2018, 74, 130- 143.
34	NAGY R , NÉMETH K , PAPP D , et al. Rootkit detection on embedded IoT devices. Acta Cybernetica, 2021, 25(2): 369- 400. doi: 10.14232/actacyb.288834
35	BRAVO P , GARCÍA D F . Rootkits survey. Architecture, 2011, 6, 7.
36	LI X Y, ZHANG Y, TANG Y. Kernel malware core implementation: a survey[C]//Proceedings of the International Conference on Cyber-Enabled Distributed Computing and Knowledge Discovery. Washington D.C., USA: IEEE Press, 2015: 9-15.
37	RUDD E M , ROZSA A , GVNTHER M , et al. A survey of stealth malware attacks, mitigation measures, and steps toward autonomous open world solutions. IEEE Communications Surveys & Tutorials, 2017, 19(2): 1145- 1172.
38	时金桥, 方滨兴, 胡铭曾, 等. Linux系统调用劫持: 技术原理、应用及检测. 计算机工程与应用, 2003, 39(32): 167- 170.
	SHI J Q , FANG B X , HU M Z , et al. Linux system call hijacking: technical principles, application and detection. Computer Engineering and Applications, 2003, 39(32): 167- 170.
39	JAKOBSSON M, RAMZAN Z. Crimeware: understanding new attacks and defenses[EB/OL]. [2023-07-05]. https://www.semanticscholar.org/paper/Crimeware:-Understanding-New-Attacks-and-Defenses-Jakobsson-Ramzan/bf54015d5373093ac73cca06d9c998c9bd53ee19.
40	PRAKASH A, VENKATARAMANI E, YIN H, et al. Manipulating semantic values in kernel data structures: attack assessments and implications[C]// Proceedings of the 43rd Annual IEEE/IFIP International Conference on Dependable Systems and Networks (DSN). Washington D.C., USA: IEEE Press, 2013: 1-12.
41	RAMASWAMY A. Detecting kernel Rootkits[EB/OL]. [2023-07-05]. http://tcipg.org/sites/default/files/papers/2008_Ramaswamy_TR2008-627.pdf.
42	刘刚, 徐峥, 崔士伟. KVM环境下内核级Rootkit检测及防护技术研究. 信息安全研究, 2019, 5(7): 616- 622.
	LIU G , XU Z , CUI S W . Research on technologies of kernel Rootkit detecting and protecting in KVM environment. Journal of Information Security Research, 2019, 5(7): 616- 622.
43	王丽娜, 高汉军, 刘炜, 等. 利用虚拟机监视器检测及管理隐藏进程. 计算机研究与发展, 2011, 48(8): 1534- 1541.
	WANG L N , GAO H J , LIU W , et al. Detecting and managing hidden process via hypervisor. Journal of Computer Research and Development, 2011, 48(8): 1534- 1541.
44	赵鲲鹏, 苏葆光. Linux内存管理中的Slab分配机制. 现代计算机, 2006(5): 89- 91.
	ZHAO K P , SU B G . Slab assigning mechanism in memory management of Linux. Modern Computer, 2006(5): 89- 91.
45	蔡梦娟. 基于VMM的虚拟机隐藏网络连接检测研究. 现代计算机, 2019(33): 15- 18.
	CAI M J . Research on virtual machine hidden network connections detection based on VMM. Modern Computer, 2019(33): 15- 18.
46	文伟平, 陈夏润, 杨法偿. 基于Rootkit隐藏行为特征的Linux恶意代码取证方法. 信息网络安全, 2020(11): 32- 42.
	WEN W P , CHEN X R , YANG F C . Malicious code forensics method based on hidden behavior characteristics of Rootkit on Linux. Netinfo Security, 2020(11): 32- 42.
47	PALAVICINI G. Bridging the detection gap: a study on a behavior-based approach using malware techniques[EB/OL]. [2023-07-05]. https://www.semanticscholar.org/paper/Bridging-the-detection-gap%3A-a-study-on-a-approach-Palavicini/d1918feb5b7459d832bb4aec31496fce39bd0352.
48	乌云, 李平, 李勇钢. 基于虚拟机自省的隐藏文件检测方法. 计算机系统应用, 2016, 25(1): 175- 180.
	WU Y , LI P , LI Y G . Method of hidden file detection based on virtual machine introspection. Computer Systems & Applications, 2016, 25(1): 175- 180.
49	BAR M. Kernel korner[EB/OL]. [2023-07-05]. https://dl.acm.org/doi/fullHtml/10.5555/348902.349094.

Please choose a citation manager

Content to export