Design and Implementation of a CXL Memory Pool System Based on gem5

doi:10.19678/j.issn.1000-3428.0068707

Abstract

Abstract:

With the advent of the era of big data, the demand for large-scale data storage and high-performance computing in data center applications is rapidly increasing. This growing need has made the access cost of massive data a significant bottleneck affecting application performance. The emergence of the Compute Express Link (CXL) interconnection protocol offers a promising solution to this challenge. This study introduces a design for a CXL extended memory pool. At the hardware level, a CXL extended memory pool system using the CXL extended memory protocol is implemented in gem5. When device memory is exposed to the CPU address space, the CPU can directly access this memory using standard load/store instructions. At the operating system level, the study develops a CXL device driver, which provides a comprehensive software stack for managing and accessing the device. In addition, utilizing the memkind library in user mode, the study integrates host and device memory to deliver a unified memory view to applications. The study builds a complete prototype of the CXL extended memory pool system based on the full system mode of gem5 and conducts a thorough evaluation of its performance. The study also compares the latency and bandwidth of host local Dynamic Random Access Memory (DRAM) and Host-managed Device Memory (HDM) using the membench and STREAM benchmarks. Experimental results show that the latency of HDM is approximately 1.5 times that of DRAM, whereas under various application scenarios, the bandwidth of HDM ranges from 50% to 63% of that of DRAM. Simultaneously, this study runs the key-value storage engine known as Viper on both DRAM and HDM and finds that in scenarios with constrained DRAM capacity, the use of extended HDM can significantly enhance system performance by a factor of 2 to 7 times.

Key words: gem5 simulator, Linux driver, Compute Express Link (CXL), memory pool, data center

摘要：

大数据时代的各类数据中心应用程序对大规模数据的存储与计算需求越来越大, 海量数据的访问开销成为限制应用程序性能的主要瓶颈, 计算快速链路(CXL)互联协议的出现为这一问题提供了新的解决思路。提出一种基于CXL的内存池系统软硬件设计。在硬件层面, 基于CXL扩展内存协议, 在系统结构模拟器gem5上构建CXL扩展内存设备。通过将设备内存暴露在中央处理器(CPU)地址空间内, 使得CPU可以直接使用load/store指令访问设备内存。在操作系统层面, 编写CXL设备的驱动程序, 为管理和访问设备提供了完整的软件栈。通过在用户态使用memkind库整合主机与设备内存, 从而向应用程序提供统一的内存视图。通过gem5的全系统模式搭建完整的CXL扩展内存池原型系统, 对系统进行全面的性能评估。使用基准测试membench和STREAM对主机本地动态随机存取内存(DRAM)与主机管理设备内存(HDM)进行了延迟和带宽的对比分析, 实验结果显示: HDM延迟约为DRAM的1.5倍, HDM的带宽约为DRAM的50%~63%。此外, 在DRAM和HDM上运行了真实的键值存储引擎Viper, 发现在DRAM容量受限的场景下, 使用扩展的HDM有2~7倍的性能提升。

关键词: gem5模拟器, Linux驱动, 快速计算链路, 内存池, 数据中心

MENG Fanfeng, WANG Zicong, ZHANG Jintao, WANG Yanjing, OU Yang, WU Lizhou, XIAO Nong. Design and Implementation of a CXL Memory Pool System Based on gem5[J]. Computer Engineering, 2025, 51(3): 180-188.

孟凡丰, 王子聪, 张金涛, 王彦景, 欧洋, 吴利舟, 肖侬. 基于gem5的CXL内存池系统设计与实现[J]. 计算机工程, 2025, 51(3): 180-188.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068707

https://www.ecice06.com/EN/Y2025/V51/I3/180

Figures/Tables 14

Fig.1 Comparison pyramid of layered access latency in the storage system

Fig.2 Software and hardware architecture of the memory pool system based on CXL

Fig.3 CXL extended memory device model

Fig.4 CXL extended memory device driver structure

Fig.5 Handling of CXL extended memory sub-protocol

Fig.6 Device enumeration log during Linux system boot

Fig.7 Configuration information of CXL devices in the system

Fig.8 Comparison of latency test results between DRAM and HDM in membench

Fig.9 Comparison of bandwidth test results between DRAM and HDM in STREAM

Fig.10 Latency test results for CXL, PMEM, and RDMA

Fig.11 Bandwidth test results for HDM and PMEM

Fig.12 QPS for four operations of a KVS running on DRAM and HDM with 2 GB of memory

Fig.13 QPS for four operations of a KVS running on DRAM and HDM with 48 MB of memory

References 27

1	KIM S, JIN Y, SOHN G, et al. Behemoth: a flash-centric training accelerator for extreme-scale DNNs[EB/OL]. [2023-09-08]. https://www.usenix.org/system/files/fast21-kim.pdf.
2	KWON Y, RHU M. Beyond the memory wall: a case for memory-centric HPC system for deep learning[C]//Proceedings of the 51st Annual IEEE/ACM International Symposium on Microarchitecture (MICRO). Washington D.C., USA: IEEE Press, 2018: 148-161.
3	SUHAS S, HARSHA S, SRAJAN G. BLAS-on-flash: an efficient alternative for large scale ML training and inference[C]//Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation. Washington D.C., USA: IEEE Press, 2019: 469-484.
4	MongoDB[EB/OL]. [2023-09-08]. https://www.mongodb.com.
5	Redis[EB/OL]. [2023-09-08]. https://www.redis.com.
6	AMARO E, BRANNER-AUGMON C, LUO Z H, et al. Can far memory improve job throughput?[C]//Proceedings of the 15th European Conference on Computer Systems. New York, USA: ACM Press, 2020: 1-16.
7	GU J C, LEE Y M, ZHANG Y W, et al. Efficient memory disaggregation with INFINISWAP[C]//Proceedings of the 14th USENIX Conference on Networked Systems Design and Implementation. [S. l. ]: USENIX Association, 2017: 649-667.
8	SHAN Y Z, HUANG Y T, CHEN Y L, et al. LegoOS: a disseminated, distributed OS for hardware resource disaggregation[C]//Proceedings of the 13th USENIX Symposium on Operating Systems Design and Implementation. [S. l. ]: USENIX Association, 2018: 69-87.
9	舒继武, 陈游旻, 汪庆, 等. 分离式数据中心的存储系统研究进展. 中国科学(信息科学), 2023, 53(8): 1503- 1528.
	SHU J W, CHEN Y M, WANG Q, et al. Progress on storage systems for disaggregated data centers. Scientia Sinica (Informationis), 2023, 53(8): 1503- 1528.
10	段卓辉, 刘海坤, 赵金玮, 等. 一种可动态配置的分布式内存池缓存一致性机制. 计算机研究与发展, 2023, 60(9): 1960- 1972.
	DUAN Z H, LIU H K, ZHAO J W, et al. A dynamically configurable distributed memory pool caching consistency mechanism. Journal of Computer Research and Development, 2023, 60(9): 1960- 1972.
11	TSAI S Y, SHAN Y Z, ZHANG Y Y. Disaggregating persistent memory and controlling them remotely: an exploration of passive disaggregated key-value stores[C]//Proceedings of 2020 USENIX Annual Technical Conference. [S. l. ]: USENIX Association, 2020: 33-48.
12	GUO Z Y, SHAN Y Z, LUO X H, et al. Clio: a hardware-software co-designed disaggregated memory system[C]//Proceedings of the 27th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York, USA: ACM Press, 2022: 417-433.
13	陆平静, 董德尊, 赖明澈, 等. 高性能计算和数据中心融合网络研究综述. 国防科技大学学报, 2023, 45(4): 1- 10.
	LU P J, DONG D Z, LAI M C, et al. Survey on converged networks of high-performance computing network and data center network. Journal of National University of Defense Technology, 2023, 45(4): 1- 10.
14	WANG Z X, SIM J, LIM E, et al. Enabling efficient large-scale deep learning training with cache coherent disaggregated memory systems[C]//Proceedings of the IEEE International Symposium on High-Performance Computer Architecture (HPCA). Washington D.C., USA: IEEE Press, 2022: 126-140.
15	Gen-Z Consortium. Gen-Z final specifications[EB/OL]. [2023-09-08]. https://genzconsortium.org/specifications/.
16	CCIX Consortium. CCIX base specification 1.1[EB/OL]. [2023-09-08]. https://www.ccixconsortium.com/library/specification/.
17	CXL Consortium. Compute Express LinkTM(CXLTM) specification[EB/OL]. [2023-09-08]. https://www.computeexpresslink.org/.
18	SHARMA D D. A low-latency and low-power approach for coherency and memory protocols on PCI express 6.0 PHY at 64.0 GT/s with PAM-4 signaling. IEEE Micro, 2022, 42(2): 37- 43. doi: 10.1109/MM.2021.3137807
19	WANG C J, HE K, FAN R Q, et al. CXL over Ethernet: a novel FPGA-based memory disaggregation design in data centers[C]//Proceedings of the IEEE 31st Annual International Symposium on Field-Programmable Custom Computing Machines (FCCM). Washington D.C., USA: IEEE Press, 2023: 75-82.
20	GOUK D, LEE S, KWON M, et al. Direct access, high-performance memory disaggregation with DirectCXL[C]//Proceedings of 2022 USENIX Annual Technical Conference. [S. l. ]: USENIX Association, 2022: 287-294.
21	AHN M, CHANG A, LEE D H, et al. Enabling CXL memory expansion for in-memory database management systems[C]//Proceedings of the International Workshop on Data Management on New Hardware. New York, USA: ACM Press, 2022: 1-5.
22	MARUF H A, WANG H, DHANOTIA A, et al. TPP: transparent page placement for CXL-enabled tiered-memory[C]//Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York, USA: ACM Press, 2023: 742-755.
23	LI H C, BERGER D S, HSU L, et al. Pond: CXL-based memory pooling systems for cloud platforms[C]//Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York, USA: ACM Press, 2023: 574-587.
24	YANG Q R, JIN R Y, DAVIS B, et al. Performance evaluation on CXL-enabled hybrid memory pool[C]//Proceedings of the IEEE International Conference on Networking, Architecture and Storage (NAS). Washington D.C., USA: IEEE Press, 2022: 1-5.
25	SHARMA D D. Compute Express Link (CXL): enabling heterogeneous data-centric computing with heterogeneous memory hierarchy. IEEE Micro, 2023, 43(2): 99- 109. doi: 10.1109/MM.2022.3228561
26	HOEFLER T, DI GIROLAMO S, TARANOV K, et al. sPIN: high-performance streaming processing in the Network[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis. Washington D.C., USA: IEEE Press, 2017: 1-16.
27	YE C C, XU Y C, SHEN X P, et al. SpecPMT: speculative logging for resolving crash consistency overhead of persistent memory[C]//Proceedings of the 28th ACM International Conference on Architectural Support for Programming Languages and Operating Systems. New York, USA: ACM Press, 2023: 762-777.

[1]	ZANG Di, YANG Zhigang, WANG Jing, YAO Zhicheng, ZHANG Weigong. Design of High-Performance Container Network Based on Network Interface Card Virtualization [J]. Computer Engineering, 2022, 48(7): 214-219.
[2]	DU Fanjie, LI Jing, GUO Zhiyong, REN Yingwen, YIN Xiaoyu, DONG Xiaoling. Coflow Scheduling Mechanism for Multi-Level Feedback Queue Based on Bottleneck Perception [J]. Computer Engineering, 2022, 48(10): 193-201,211.
[3]	ZUO Pan, SHU Yongan. Dynamic Multi-Path Load Balancing Method Based on Feedforward Neural Network in DCN [J]. Computer Engineering, 2021, 47(9): 113-119.
[4]	QIAN Shengpan, YU Yang, ZHAI Tianyi, ZHANG Xudong, CHANG Yanbo. Research on Online Prediction Model of Host Load Based on Deep Learning [J]. Computer Engineering, 2021, 47(9): 84-89.
[5]	FAN Yuqi, ZHANG Bei, WANG Lunfei. Research on Data Copy Placement for Improved Power Consumption and Network Synchronization Cost [J]. Computer Engineering, 2020, 46(2): 110-117.
[6]	XU Chao, DENG Junhua, JIANG Lili, JI Mingtao, LI Xin. QoS-Aware Data Replica Recovery Algorithm for Cloud System [J]. Computer Engineering, 2019, 45(9): 17-22.
[7]	ZANG Weifei,LAN Julong,HU Yuxiang,CHEN Wentao. Network-aware virtual machine placement algorithm based on multi-objective discrete differential evolution [J]. Computer Engineering, 2019, 45(6): 96-102.
[8]	ZHANG Zhao,LI Hailong,DONG Siqi,HU Lei,MA Jingren. SDN-based Flow Probability Path Selection Method for Data Center Network [J]. Computer Engineering, 2019, 45(4): 36-40.
[9]	LIU Kainan. Virtual Machine Selection Strategy Based on Task Mapping for Cloud Data Center [J]. Computer Engineering, 2019, 45(10): 33-39.
[10]	LI Wenxin, ZHOU Xiaobo, XU Renhai, QI Heng, LI Keqiu. An Approximate Smallest-Effective-Bottleneck-First Coflow Scheduling Mechanism [J]. Computer Engineering, 2019, 45(10): 19-25,32.
[11]	LI Youming,HUANG Jiawei,XU Wenxi,WANG Jianxin. Task-aware Transmission Control Protocol in Data Center Network [J]. Computer Engineering, 2018, 44(4): 145-148,153.
[12]	GUO Yueyue,HUANG Jiawei,LIU Jingling,WANG Jianxin. Random Backoff Method for Highly Concurrent TCP Transmission in Data Center Network [J]. Computer Engineering, 2018, 44(4): 135-139,144.
[13]	DING Jianli,ZHOU Dongxue,WANG Jing,WANG Jialiang. Remote Database Synchronization Mechanism Based on SQL Capture from Connection Driver [J]. Computer Engineering, 2017, 43(9): 39-42,50.
[14]	ZHANG Junwei,WANG Jing,ZHANG Weigong,QIU Keni. Research of High Energy-efficient Data Center Server Based on Big Data [J]. Computer Engineering, 2017, 43(8): 74-81.
[15]	WANG Xiaojie,XU Mingwei,WANG Sixiu,ZHU Yixin. Two-phase Virtual Machine Placement Algorithm Based on Network Awareness [J]. Computer Engineering, 2017, 43(8): 32-37.

Please choose a citation manager

Content to export