一种集成于超算作业调度系统应用的并行参数优化方法

doi:10.19678/j.issn.1000-3428.0069035

摘要/Abstract

摘要：

随着高性能计算体系结构的发展，软件与硬件都具有多层的并行结构。当不同纵向层级与横向分组的计算任务被划分到不同节点的不同处理器时，存在非常多的分配方式。这些分配方式一般在运行时由用户输入的多个并行参数来确定，并对计算效率影响很大。随着计算规模与复杂度的提升，多个并行参数的可配置空间越来越大，用户越来越难以确定最佳的并行参数值。这类运行时优化问题在科学计算应用中较为普遍，但相关的研究与解决方法比较少见。以VASP(Vienna Ab initio Simulation Package)应用为例，首先分析了该应用的多层并行结构，展示了不同并行参数配置引发的巨大运行速度差异。然后提出了一个基于约化并行效率指标的全自动运行优化方法，其不仅可以帮助用户简单快捷地确定最佳应用并行参数，而且可以帮助用户确定最佳的计算资源使用量，使应用可以高效率地扩展到大规模的并行计算中。最后将该优化方法与计算集群作业调度系统相融合应用于用户提交的真实VASP计算作业。统计结果表明，该方法显著提升了作业运行速度与超算资源的使用效率，具有很好的工程应用前景。

关键词: 并行计算, 作业调度, 运行时优化, 超级计算, VASP应用

Abstract:

High-performance computing architectures have facilitated software and hardware with multi-layer parallel structures. These multi-layered system resources can be assigned to computational tasks distributed across different vertical tiers and horizontal groups through various schemes. The allocation schemes, typically determined at runtime by user-defined parallel parameters, significantly affect computational efficiency. As computational scale and complexity increase, the configurable space for these parallel parameters expands, making it increasingly difficult for users to identify the optimal settings. Although such runtime optimization problems are prevalent in scientific computing applications, related research and effective solutions remain scarce. Using the Vienna Ab initio Simulation Package (VASP) as a case study, this study to analyze its multilayer parallel structure to demonstrate how different parallel parameter configurations can lead to significant variations in computational speed. It then proposes a fully automated optimization method based on a reduced parallel efficiency metric. This approach enables users to quickly determine optimal parallel parameters and identifies the most efficient hardware resource allocation, facilitating effective scaling for large-scale parallel computing. Finally, this study integrates the optimization method with a cluster job scheduling system and applies it to actual VASP calculation jobs submitted by users. Statistical results demonstrate that the proposed method significantly improves job execution speed and enhances the utilization efficiency of supercomputing resources, showing great promise for practical engineering applications.

Key words: parallel computing, job scheduling, runtime optimization, supercomputing, VASP application

张文帅, 李会民, 李京, 潘必才. 一种集成于超算作业调度系统应用的并行参数优化方法[J]. 计算机工程, 2025, 51(7): 59-67.

ZHANG Wenshuai, LI Huimin, LI Jing, PAN Bicai. A Parallel Parameter Optimization Method Integrated with Job Scheduling System for Supercomputing Applications[J]. Computer Engineering, 2025, 51(7): 59-67.

https://www.ecice06.com/CN/Y2025/V51/I7/59

图/表 7

图1 VASP应用中同时运行的任务的并行划分结构

Fig.1 Parallel division structure of tasks running simultaneously in a VASP applicaiton

图2 在根据经验缩小的空间中进行的参数遍历测试结果

Fig.2 Parameter traversal test results conducted in a space narrowed down based on experience

图3 作业调度系统与并行参数优化方法的集成流程

Fig.3 Integrated process of job scheduling system and parallel parameter optimization method

图4 VASP程序运行时自动优化效果

Fig.4 Automatic optimization effect of VASP program runtime

参考文献 32

1	MARTORELL X, AYGUADÉ E, NAVARRO N, et al. Thread fork/join techniques for multi-level parallelism exploitation in NUMA multiprocessors[C]//Proceedings of the 13th International Conference on Supercomputing. New York, USA: ACM Press, 1999: 284-301.
2	LU K , WANG Y H , GUO Y , et al. MT-3000:a heterogeneous multi-zone processor for HPC. CCF Transactions on High Performance Computing, 2022, 4 (2): 150- 164. doi: 10.1007/s42514-022-00095-y
3	DONGARRA J , STERLING T , SIMON H , et al. High-performance computing: clusters, constellations, MPPs, and future directions. Computing in Science & Engineering, 2005, 7 (2): 51- 59. doi: 10.1109/MCSE.2005.34
4	DIAZ J , MUÑOZ-CARO C , NIÑO A . A survey of parallel programming models and tools in the multi and many-core era. IEEE Transactions on Parallel and Distributed Systems, 2012, 23 (8): 1369- 1386. doi: 10.1109/TPDS.2011.308
5	HACKENBERG D , JUCKELAND G , BRUNST H . Performance analysis of multi-level parallelism: inter-node, intra-node and hardware accelerators. Concurrency and Computation: Practice and Experience, 2012, 24 (1): 62- 72. doi: 10.1002/cpe.1725
6	涂碧波, 邹铭, 詹剑锋, 等. 多核处理器机群Memory层次化并行计算模型研究. 计算机学报, 2008, 31 (11): 1948- 1955. doi: 10.3321/j.issn:0254-4164.2008.11.009
	TU B B , ZOU M , ZHAN J F , et al. Research on parallel computation model with memory hierarchy on multi-core clusters. Chinese Journal of Computers, 2008, 31 (11): 1948- 1955. doi: 10.3321/j.issn:0254-4164.2008.11.009
7	张云泉, 袁良, 陈一峯, 等. 高性能计算多层次不连续非线性可扩展现象研究. 计算机学报, 2020, 43 (6): 973- 989. doi: 10.11897/SP.J.1016.2020.00973
	ZHANG Y Q , YUAN L , CHEN Y F , et al. Multi-level discontinuous and nonlinear scalability phenomenon in high performance computing. Chinese Journal of Computers, 2020, 43 (6): 973- 989. doi: 10.11897/SP.J.1016.2020.00973
8	臧大伟, 曹政, 孙凝晖. 高性能计算的发展. 科技导报, 2016, 34 (14): 22- 28.
	ZANG D W , CAO Z , SUN N H . The development of high-performance computing. Science & Technology Review, 2016, 34 (14): 22- 28.
9	金钟, 陆忠华, 李会元, 等. 高性能计算之源起——科学计算的应用现状及发展思考. 中国科学院院刊, 2019, 34 (6): 625- 639.
	JIN Z , LU Z H , LI H Y , et al. Origin of high performance computing-current status and developments of scientific computing applications. Bulletin of Chinese Academy of Sciences, 2019, 34 (6): 625- 639.
10	王涛. 量子化学中的高性能计算. 华东师范大学学报(自然科学版), 2018 (4): 109- 119.
	WANG T . High performance computing in quantum chemistry. Journal of East China Normal University (Natural Science), 2018 (4): 109- 119.
11	GARCIA-GASULLA M , BANCHELLI F , PEIRO K , et al. A generic performance analysis technique applied to different CFD methods for HPC. International Journal of Computational Fluid Dynamics, 2020, 34 (7/8): 508- 528. doi: 10.1080/10618562.2020.1778168
12	GAVIN J P. Optimisation of VASP Density Functional Theory (DFT)/Hartree-Fock (HF) hybrid functional code using mixed-mode parallelism[EB/OL]. [2023-11-20]. http://www.hector.ac.uk/cse/distributedcse/reports/vasp02/vasp02.pdf.
13	CATLOW R, WOODLEY S, DE LEEUW N, et al. Optimising the performance of the VASP 5.2.2 code on HECToR[EB/OL]. [2023-11-20]. http://www.hector.ac.uk/cse/distributedcse/reports/vasp01/vasp01_collectives.pdf.
14	RASCH A , SCHULZE R , STEUWER M , et al. Efficient auto-tuning of parallel programs with interdependent tuning parameters via Auto-Tuning Framework (ATF). ACM Transactions on Architecture and Code Optimization, 2021, 18 (1): 1- 26. doi: 10.1145/3427093
15	MENON H, BHATELE A, GAMBLIN T. Auto-tuning parameter choices in HPC applications using Bayesian optimization[C]//Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS). Washington D.C., USA: IEEE Press, 2020: 831-840.
16	FRIGO M, JOHNSON S G. FFTW: an adaptive software architecture for the FFT[C]//Proceedings of the 1998 IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 1998: 1381-1384.
17	GU L, LI X M. DFT performance prediction in FFTW[C]//Proceedings of the 22nd International Workshop on Languages and Compilers for Parallel Computing. Berlin, Germany: Springer, 2010: 140-156.
18	GUARRASI M, ERBACCI G, EMERSON A. Auto-tuning of the FFTW library for massively parallel supercomputers[EB/OL]. [2023-11-20]. https://inspirehep.net/literature/2801065.
19	BECKINGSALE D, PEARCE O, LAGUNA I, et al. Apollo: reusable models for fast, dynamic tuning of input-dependent code[C]//Proceedings of the IEEE International Parallel and Distributed Processing Symposium (IPDPS). Washington D.C., USA: IEEE Press, 2017: 307-316.
20	MATTSON T G, CLEDAT R, CAVÉ V, et al. The Open Community Runtime: a runtime system for extreme scale computing[C]//Proceedings of the IEEE High Performance Extreme Computing Conference (HPEC). Washington D.C., USA: IEEE Press, 2016: 1-7.
21	DOKULIL J, SANDRIESER M, BENKNER S. OCR-Vx-an alternative implementation of the open community runtime[C]//Proceedings of the International Workshop on Runtime Systems for Extreme Scale Programming Models and Architectures in Conjunction with SC15. Washington D.C., USA: IEEE Press, 2015: 1-10.
22	DOKULIL J, BENKNER S. Automatic placement of tasks to NUMA nodes in iterative applications[C]//Proceedings of the 28th Euromicro International Conference on Parallel, Distributed and Network-Based Processing. Washington D.C., USA: IEEE Press, 2020: 192-195.
23	KRESSE G , FURTHMVLLER J . Efficiency of ab-initio total energy calculations for metals and semiconductors using a plane-wave basis set. Computational Materials Science, 1996, 6 (1): 15- 50. doi: 10.1016/0927-0256(96)00008-0
24	GIANNOZZI P , BARONI S , BONINI N , et al. QUANTUM ESPRESSO: a modular and open-source software project for quantum simulations of materials. Journal of Physics Condensed Matter, 2009, 21 (39): 395502. doi: 10.1088/0953-8984/21/39/395502
25	STEGAILOV V , SMIRNOV G , VECHER V . VASP hits the memory wall: processors efficiency comparison. Concurrency and Computation: Practice and Experience, 2019, 31 (19): 5136. doi: 10.1002/cpe.5136
26	STEGAILOV V , VECHER V . Efficiency analysis of intel and AMD x86_64 architectures for ab initio calculations: a case study of VASP. Berlin, Germany: Springer, 2017.
27	ZHAO Z J, MARSMAN M, WENDE F, et al. Performance of hybrid MPI/OpenMP VASP on cray XC40 based on intel knights landing many integrated core architecture[EB/OL]. [2023-11-20]. https://www.semanticscholar.org/paper/Performance-of-Hybrid-MPI-OpenMP-VASP-on-Cray-XC40-Zhao-Marsman/51bb2d68a57b7cf5fc627194d06f716c7832fb26.
28	MCINTOSH-SMITH S , PRICE J , DEAKIN T , et al. A performance analysis of the first generation of HPC-optimized arm processors. Concurrency and Computation: Practice and Experience, 2019, 31 (16): 5110. doi: 10.1002/cpe.5110
29	WU G B , SHEN Y , ZHANG W S , et al. Runtime prediction of jobs for backfilling optimization. Journal of Chinese Computer Systems, 2019, 40 (1): 6- 12. URL
30	WANG Y R, DU Z H, JIANG J, et al. Modeling the parallel efficiency of density functional theory based jobs on sunway TaihuLight[C]//Proceedings of the IEEE International Conference on Computational Science and Engineering (CSE) and IEEE International Conference on Embedded and Ubiquitous Computing (EUC). Washington D.C., USA: IEEE Press, 2019: 199-204.
31	ZHANG W S. Previous benchmark for VASP[EB/OL]. [2023-11-20]. https://scc.ustc.edu.cn/_upload/article/files/cd/05/bfefd0ec4fc08d2843fded77bca5/3118fa63-8f2d-4fd5-8582-161dfe1342e8.pdf.
32	GEUN H G. It is wise to do your own test to optimize this parameter[EB/OL]. [2023-11-20]. https://www.researchgate.net/post/NCORE-and-NPAR-in-VASP.

[1]	潘顺杰, 于俊洋, 王龙葛, 李涵, 翟锐. 基于RDD重用度的Spark自适应缓存优化策略[J]. 计算机工程, 2025, 51(7): 190-198.
[2]	杨太龙, 赵红朋, 张磊. 基于国产异构平台的奇异值分解法[J]. 计算机工程, 2024, 50(9): 216-225.
[3]	张磊, 赵光岳, 肖超恩, 王建新. Falcon后量子算法的密钥树生成部件GPU并行优化设计与实现[J]. 计算机工程, 2024, 50(9): 208-215.
[4]	雷斗威, 何德彪, 罗敏, 彭聪. 基于AVX512的格密码高速并行实现[J]. 计算机工程, 2024, 50(2): 15-24.
[5]	王其涵, 庞建民, 岳峰, 祝迪, 沈莉, 肖谦. 面向申威架构的KNN并行算法实现与优化[J]. 计算机工程, 2023, 49(5): 286-294.
[6]	夏立斌, 刘晓宇, 姜晓巍, 孙功星. 基于分布式数据集的并行计算框架内存优化方法[J]. 计算机工程, 2023, 49(4): 43-51.
[7]	房俊, 薛晓东, 周云亮. 基于深度生成模型的聚合查询区间估计方法[J]. 计算机工程, 2023, 49(11): 284-292, 301.
[8]	喻志超, 李扬中, 刘磊, 冯圣中. 量子计算模拟及优化方法综述[J]. 计算机工程, 2022, 48(1): 1-11.
[9]	黄瑞, 金光浩, 李磊, 姜文超, 宋庆增. 轻量化神经网络加速器的设计与实现[J]. 计算机工程, 2021, 47(9): 185-190,196.
[10]	易培淮, 李卫东, 林韬, 邹佳恒, 邓子艳, 刘言. GPU在缪子快速模拟中的应用[J]. 计算机工程, 2021, 47(8): 100-108.
[11]	佘鑫, 何震瀛. 复杂属性条件下基于Spark的clique社区搜索算法[J]. 计算机工程, 2021, 47(12): 54-61,70.
[12]	郭渝洛, 边浩东, 董润婷, 唐嘉豪, 王晓英, 黄建强. 基于SIMD的并行傅里叶空间图像相似度计算[J]. 计算机工程, 2021, 47(11): 247-253.
[13]	肖成龙, 聂紫阳, 王宁, 张重鹏, 王珊珊. 基于并行约束规划的最大团识别研究[J]. 计算机工程, 2020, 46(4): 53-59,69.
[14]	李洁, 朱洪亮, 陈玉玲, 辛阳. 基于哈希存储与事务加权的并行Apriori改进算法[J]. 计算机工程, 2020, 46(11): 109-116.
[15]	徐国伟, 陈建, 成怡. 基于GPU并行计算的雷达杂波模拟研究[J]. 计算机工程, 2020, 46(11): 306-314.

选择文件类型/文献管理软件名称

选择包含的内容