应用透明的超算多层存储加速技术研究

doi:10.19678/j.issn.1000-3428.0065928

摘要/Abstract

摘要： 在E级计算时代，超算系统一般使用多层存储架构以满足应用数据访问的容量和性能需求，这种架构中不同层次的存储介质差异较大，难以实现统一名字空间管理，往往需要应用修改数据访问流程才能最大程度利用到多层存储的性能和容量优势。针对多层存储统一名字空间的问题，提出针对非易失性双列存储模块（NVDIMM）的块级缓存和针对突发缓冲存储（BB）的文件级缓存技术。基于NVDIMM的块级缓存技术对缓存窗口灵活控制，以支持数据块粒度的异步读写，实现NVDIMM与BB层统一名字空间管理；基于BB的文件级缓存技术将数据缓存在BB层中，并动态迁移和管理文件副本，实现BB层与传统磁盘文件系统统一名字空间管理。在神威E级原型验证系统中的测试结果表明，所提出的两种技术较好地解决了多层存储的透明加速难题，NVDIMM块级缓存与BB相比，在缓存窗口16 MB时128 KB顺序读写带宽分别提升27%和36%，8 KB随机读写带宽分别提升20%和37%；基于BB的文件缓存技术利用BB的高带宽支撑数据访问，与全局文件系统相比，128 KB顺序读写带宽分别提升55%和141%，8 KB随机读写带宽分别提升163%和209%。此外，实际应用的测试也表明以上两种缓存技术具有透明的存储加速效果。

关键词: 超算系统, 分层存储, 非易失性双列存储模块, 突发缓冲存储, 块级缓存, 文件级缓存, 透明加速

Abstract: Supercomputing systems have entered the exascale era and to meet the growing demand for data access and performance, supercomputing systems typically require multi-tier storage architecture.Storage media typically vary between tiers, making it difficult to achieve unified namespace management.This paper proposes to address this problem by developing block-level caching based on Non-Volatile Dual In-line Memory Module(NVDIMM) and file-level cache based on Burst Buffer(BB) technologies.The NVDIMM-based block-level cache supports asynchronous read/write of data block granularity through flexible control of the cache window and realizes the unified namespace of NVDIMM and BB.The file-level cache is based on BB and supports data caching in it, dynamically migrates and manages file copies, and realizes the unified namespace between BB and the traditional disk file system.Evaluation result of the verification systems in the Sunway E-class prototype shows that both technologies proposed in this paper can achieve transparent acceleration.Compared with BB, the NVDIMM block-level cache increases the sequential read/write bandwidth of a 128 KB block by 27% and 36%, respectively, and the random read/write bandwidth of an 8 KB block by 20% and 37%, respectively, given a cache window size of 16 MB.The BB-based file cache allows applications to exploit the high bandwidth of BB.Compared with the Global File System(GFS), the 128 KB block sequential read/write bandwidth increases by 55% and 141%, respectively, and the 8 KB random read/write bandwidth increases by 163% and 209%, respectively.In addition, the practical application presented in this study shows that above two cache technologies have transparent storage acceleration effects.

Key words: supercomputing system, tiered storage, Non-Volatile Dual In-line Memory Module(NVDIMM), Burst Buffer(BB), block-level cache, file-level cache, transparent acceleration

中图分类号:

TP333

何晓斌, 高洁, 肖伟, 陈起, 刘鑫, 陈左宁. 应用透明的超算多层存储加速技术研究[J]. 计算机工程, 2022, 48(12): 1-8.

HE Xiaobin, GAO Jie, XIAO Wei, CHEN Qi, LIU Xin, CHEN Zuoning. Research on Application-Transparent Supercomputing Multi-tier Storage Acceleration Technology[J]. Computer Engineering, 2022, 48(12): 1-8.

http://www.ecice06.com/CN/Y2022/V48/I12/1

图/表 11

20230112182022

20230112182026

20230112182030

20230112182034

20230112182038

20230112182041

20230112182045

20230112182049

20230112182053

20230112182056

20230112182100

参考文献

[1] BOITO F Z, INACIO E C, BEZ J L, et al.A checkpoint of research on parallel I/O for high-performance computing[J].ACM Computing Surveys, 2019, 51(2):23.
[2] BHARATHI S, CHERVENAK A, DEELMAN E, et al.Characterization of scientific workflows[C]//Proceedings of the 3rd Workshop on Workflows in Support of Large-Scale Science.Washington D.C., USA:IEEE Press, 2008:1-10.
[3] YASHIRO H, TERASAKI K, KAWAI Y T, et al.A 1024-member ensemble data assimilation with 3.5-km mesh global weather simulations[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis.Washington D.C., USA:IEEE Press, 2020:1-10.
[4] LIU N, COPE J, CARNS P, et al.On the role of Burst Buffers in leadership-class storage systems[C]//Proceedings of IEEE Symposium on Mass Storage Systems and Technologies.Washington D.C., USA:IEEE Press, 2016:1-11.
[5] PATIL O, IONKOV L, LEE J, et al.Performance characterization of a DRAM-NVM hybrid memory architecture for HPC applications using intel optane DC persistent memory modules[C]//Proceedings of International Symposium on Memory Systems.New York, USA:ACM Press, 2019:1-5.
[6] Oak Ridge National Laboratory.Frontier[EB/OL].(2022-05-30)[2022-09-10].https://www.ornl.gov/news/frontier-supercomputer-debuts-worlds-fastest-breaking-exascale-barrier.
[7] AKIMOTO H, OKAMOTO T, KAGAMI T, et al.File system and power management enhanced for supercomputer Fugaku[EB/OL].(2020-11-11)[2022-09-10].https://www.fujitsu.com/global/about/resources/publications/technicalreview/2020-03/article05.html.
[8] VEF M A, MOTI N, SÜß T, et al.GekkoFS-a temporary distributed file system for HPC applications[C]//Proceedings of IEEE International Conference on Cluster Computing.Washington D.C., USA:IEEE Press, 2018:319-324.
[9] Unifyfs.UnifyFS:a file system for Burst Buffers[EB/OL].[2022-09-10].https://unifyfs.readthedocs.io/en/latest/index.html.
[10] 陈曦, 朱建涛, 何晓斌.一种面向高性能计算的分布式对象存储系统[J].计算机工程, 2017, 43(8):69-73. CHEN X, ZHU J T, HE X B.An HPC-oriented distributed object storage system[J].Computer Engineering, 2017, 43(8):69-73.(in Chinese)
[11] DAVE H, BENJAMIN L, DOUG P.Architecture and design of cray datawarp[EB/OL].(2016-05-28)[2022-09-10].https://cug.org/proceedings/cug2016_proceedings/includes/files/pap105s2-file1.pdf.
[12] 宋振龙, 李小芳, 李琼, 等.BeeGFS并行文件系统性能优化技术研究[J].计算机工程与科学, 2020, 42(10):1765-1773. SONG Z L, LI X F, LI Q, et al.Improving the performance of BeeGFS parallel file system[J].Computer Engineering & Science, 2020, 42(10):1765-1773.(in Chinese)
[13] Thinkparq.BeeOND:BeeGFS on demand[EB/OL].[2022-09-10].https://beegfs-docs.readthedocs.io/en/latest/beeond.html.
[14] PATEL T, BYNA S, LOCKWOOD G K, et al.Uncovering access, reuse, and sharing characteristics of I/O-intensive files on large-scale production hpc systems[C]//Proceedings of the 18th USENIX Conference on File and Storage Technologies.Santa Clara, USA:USENIX Association, 2020:91-101.
[15] DONG B, BYNA S, WU K S, et al.Data elevator:low-contention data movement in hierarchical storage system[C]//Proceedings of IEEE International Conference on High Performance Computing.Washington D.C., USA:Washington D.C., USA:IEEE Press, 2016:152-161.
[16] WANG T, BYNA S, DONG B, et al.UniviStor:integrated hierarchical and distributed storage for HPC[C]//Proceedings of IEEE International Conference on Cluster Computing.Washington D.C., USA:IEEE Press, 2018:134-144.
[17] KOUGKAS A, DEVARAJAN H, SUN X H.I/O acceleration via multi-tiered data buffering and prefetching[J].Journal of Computer Science and Technology, 2020, 35(1):92-120.
[18] YANG B, JI X, MA X S, et al.End-to-end I/O monitoring on a leading supercomputer[C]//Proceedings of the 16th USENIX Conference on Networked Systems Design and Implementation.Boston, USA:USENIX Association, 2019:379-394.
[19] CHEN Q, CHEN K, CHEN Z N, et al.Lessons learned from optimizing the Sunway storage system for higher application I/O performance[J].Journal of Computer Science and Technology, 2020, 35(1):47-60.
[20] QIAN Y, WANG F, FENG D, et al.LPCC:hierarchical persistent client caching for Lustre[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis.New York, USA:ACM Press, 2019:1-14.
[21] 李春艳.面向NVM的Lustre客户端持久性缓存研究[D].武汉:华中科技大学, 2019. LI C Y.Research on NVM-oriented Lustre persistent cache on client[D].Wuhan:Huazhong University of Science and Technology, 2019.(in Chinese)
[22] LOFSTEAD J, JIMENEZ I, MALTZAHN C, et al.DAOS and friends:a proposal for an exascale storage system[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis.Washington D.C., USA:IEEE Press, 2016:585-596.
[23] 薛巍, 杨斌, 刘世超, 等.Beacon+:面向E级超级计算机的轻量级端到端I/O性能监控与分析诊断系统[C]//2021 CCF全国高性能计算学术年会大会论文集.珠海:中国计算机学会, 2021:1-12. XUE W, YANG B, LIU S C, et al.Beacon+:a scalable lightweight end-to-end I/O performance monitoring, analysis anddiagnosis system for exascale supercomputer[C]//Proceedings of 2021 CCF National Annual Conference on High Performance Computing.Zhuhai, China:CCF, 2021:1-12.(in Chinese)
[24] WANG W, DIESTELHORST S.Quantify the performance overheads of PMDK[C]//Proceedings of International Symposium on Memory Systems.New York, USA:ACM Press, 2018:50-52.
[25] 朱文俊, 徐壮, 秦家佳, 等.基于DPDK的高速存储I/O优化方法[J].计算机工程, 2021, 47(7):205-211, 217. ZHU W J, XU Z, QIN J J, et al.DPDK-based optimization approach for high speed storage I/O[J].Computer Engineering, 2021, 47(7):205-211, 217.(in Chinese)
[26] 程振京, 汪璐, 程耀东, 等.面向高能物理分级存储的文件访问热度预测[J].计算机工程, 2021, 47(2):126-132. CHENG Z J, WANG L, CHENG Y D, et al.File access popularity prediction for hierarchical storage for high-energy physics[J].Computer Engineering, 2021, 47(2):126-132.(in Chinese)
[27] BEZ J L, KARIMI A M, PAUL A K, et al.Access patterns and performance behaviors of multi-layer supercomputer I/O subsystems under production load[C]//Proceedings of the 31st International Symposium on High-Performance Parallel and Distributed Computing.New York, USA:ACM Press, 2022:1-5.
[28] ORAL S, VAZHKUDAI S S, WANG F, et al.End-to-end I/O portfolio for the summit supercomputing ecosystem[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis.New York, USA:ACM Press, 2019:1-14.
[29] 高剑刚, 卢宏生, 何王全, 等.神威E级原型机互连网络和消息机制[J].计算机学报, 2021, 44(1):222-234. GAO J G, LU H S, HE W Q, et al.The interconnection network and message machinasim of sunway exascale prototype system[J].Chinese Journal of Computers, 2021, 44(1):222-234.(in Chinese)
[30] 三台E级超算原型机系统全部完成交付[J].河南科技, 2018(31):6. Three exascale supercomputing prototype systems have been delivered[J].Henan Science and Technology, 2018(31):6.(in Chinese)
[31] FU H H, LIAO J F, YANG J Z, et al.The Sunway TaihuLight supercomputer:system and applications[J].Science China Information Sciences, 2016, 59(7):072001.
[32] LIU Y A, LIU X L, LI F N, et al.Closing the "quantum supremacy" gap:achieving real-time simulation of a random quantum circuit using a new Sunway supercomputer[C]//Proceedings of International Conference for High Performance Computing, Networking, Storage and Analysis.New York, USA:ACM Press, 2021:1-5.
[33] IOR.HPC IO benchmark repository[EB/OL].[2022-09-10].https://github.com/hpc/ior.
[34] SKAMAROCK W C, KLEMP J B, DUDHIA J.Prototypes for the WRF(Weather Research and Forecasting) model[EB/OL].(2001-01-01)[2022-09-10].https://www.researchgate.net/publication/242446432_Prototypes_for_the_WRF_Weather_Research_and_Forecasting_model.
[35] H5bench[EB/OL].[2022-09-10].https://github.com/hpc-io/h5bench.

选择文件类型/文献管理软件名称

选择包含的内容