GRAPES动力框架中大规模稀疏线性系统并行求解及优化

doi:10.19678/j.issn.1000-3428.0060080

摘要/Abstract

摘要： GRAPES(global/regional assimilation prediction system)数值天气预报系统动力框架中的求解核心之一为赫姆霍兹方程，该方程本质上可以转换为大规模稀疏线性系统的求解问题。由于硬件资源的限制及数据规模的不断增大，求解效率逐渐成为了限制计算性能提升的瓶颈。针对这个问题，通过 MPI、MPI+OpenMP、CUDA 三种并行方式实现了求解线性方程组的广义共轭余差法(GCR)。实验首先通过 ILU 预处理子改善系数矩阵的条件数以加快迭代法收敛。然后在 CPU 并行方案中，MPI 负责进程间粗粒度并行和通信，OpenMP 结合共享内存实现进程内部的细粒度并行；在 GPU 并行方案中，CUDA 模型采用了数据传输、访存合并及共享存储器方面的优化措施。最后对结果进行了准确性验证及性能分析。实验结果表明，MPI+OpenMP 混合并行优化比 MPI 并行优化性能高约 35%，CUDA 并行优化比 MPI+OpenMP 混合并行优化性能高约 50%。

Abstract: One of the core solutions in the dynamic framework of GRAPES (global/regional assimilation prediction system) numerical weather prediction system is the Helmholtz equation. This equation can essentially be transformed into the solution of a large-scale sparse linear system. Due to the limitation of hardware resources and the continuous increase of data scale, solution efficiency has gradually become a bottleneck that limits the improvement of computing performance. To solve this problem, the generalized conjugate residual method (GCR) for solving linear equations is realized through three parallel methods of MPI, MPI+OpenMP and CUDA. The experiment first improves the condition number of the coefficient matrix through the ILU preconditioner to speed up the convergence of the iterative method. Then in the CPU parallel scheme, MPI is responsible for coarse-grained parallelism and communication between processes, and OpenMP combines with shared memory to achieve fine-grained parallelism within the process; In the GPU parallel scheme, the CUDA model uses the optimization of data transmission, coalesced access and shared memory. Finally, the accuracy verification and performance analysis of the results are carried out. Experimental results show that the performance of MPI+OpenMP hybrid parallel optimization is about 35% higher than that of MPI parallel optimization, and the performance of CUDAparallel optimization is about 50% higher than that of MPI+OpenMP hybrid parallel optimization.

张琨 , 贾金芳 , 严文昕 , 黄建强 , 王晓英 . GRAPES动力框架中大规模稀疏线性系统并行求解及优化[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0060080.

Zhang Kun , Jia Jinfang , Yan Wenxin , Huang Jianqiang , Wang Xiaoying. Parallel solution and optimization of large-scale sparse linear system in GRAPES dynamic framework[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0060080.

参考文献

[1] Ahamed A K C, Magoules F. Iterative methods for sparse linear systems on graphics processing unit[C]//2012 IEEE 14th International Conference on High Performance Computing and Communication & 2012 IEEE 9th International Conference on Embedded Software and Systems. IEEE, 2012: 836-842.
[2] D'Yakonov Y G. Matrix iterative analysis: Varga, R. S., 1963[J]. USSR Computational Mathematics and Mathematical Physics, 1965, 5(2): 303-307.
[3] Zhai Yan, Zhai Jidong, Xue Wei, et al. Massive parallel program communication performance analysis [J]. Journal of Huazhong University of Science and Technology (Natural Science Edition), 2011, 39(S1): 71-75. (in Chinese) 翟琰，翟季冬，薛巍，等. 大规模并行程序通信性能分析[J]. 华中科技大学学报(自然科学版), 2011, 39(S1): 71-75.
[4] Yang Lei. Communication Avoiding Generalized Conjugate Residual Method[D]. Chinese Academy ofMeteorological Sciences, 2019. (in Chinese) 杨磊. 通信避免的广义共轭余差算法[D]. 中国气象科学研究院, 2019.
[5] Zhang Lilun. Research of High-performance Parallel Computing for Numerical Models in Meteorologic Prediction[D]. National University of Defense Technology, 2002. (in Chinese) 张理论. 面向气象预报数值模式的高效并行计算研究 [D]. 国防科学技术大学, 2002.
[6] Li X, Peng X, Li X. An improved dynamic core for a non-hydrostatic model system on the Yin-Yang grid[J]. Advances in Atmospheric Sciences, 2015, 32(5): 648-658.
[7] Jiang Qingu, Jin Zhiyan. The hybrid MPI and OpenMP Parallel Scheme of GRAPES_Global Model[J]. Journal of Applied Meteorological Science, 2014, 25(05): 581-591. (in Chinese) 蒋沁谷，金之雁. GRAPES全球模式MPI与OpenMP混合并行方案[J]. 应用气象学报, 2014, 25(05): 581-591.
[8] Liu Zhao. Study of GRAPES Numerical Weather Prediction System Optimization on Domestic High Performance Computers[D]. Shanghai Jiaotong University, 2014. (in Chinese) 刘钊. 基于国产高性能计算机的GRAPES性能优化研究[D]. 上海交通大学, 2014.
[9] Wu Xiangjun. Study on the Parallel Computing in GRAPES High Resolution Numerical Weather Prediction Model[D]. National University of Defense Technology, 2011. (in Chinese) 伍湘君. GRAPES高分辨率气象数值预报模式并行计算关键技术研究[D]. 国防科学技术大学, 2011.
[10] Huang J, Xue W, Bian H, et al. Helmholtz solving and performance optimization in global/regional assimilation and prediction system[J]. Tsinghua Science and Technology, 2021, 26(3): 335-346.
[11] Wenxin Y, Jinfang J, Jianqiang H, et al. Research of GRAPES Numerical Weather Prediction Model[C]//Proceedings of the 2020 4th High Performance Computing and Cluster Technologies Conference & 2020 3rd International Conference on Big Data and Artificial Intelligence. 2020: 34-41.
[12] Zou Y, Xue W, Liu S. A case study of large-scale parallel I/O analysis and optimization for numerical weather prediction system[J]. Future Generation Computer Systems, 2014, 100(37): 378-389.
[13] Shen X, Wang J, Li Z, et al. Research and Operational Development of Numerical Weather Prediction in China[J]. Journal of Meteorological Research, 2020, 34(4): 675-698.
[14] Chen D, Xue J, Yang X, et al. New generation of multi-scale NWP system (GRAPES): general scientific design[J]. Chinese Science Bulletin, 2008, 53(22): 3433-3445
[15] Xu G, Chen D, Xue J, et al. The program structure designing and optimizing tests of GRAPES physics[J]. Chinese Science Bulletin, 2008, 53(22): 3470-3476.
[16] Huang B, Chen D, Li X, et al. Improvement of the semi-Lagrangian advection scheme in the GRAPES model: Theoretical analysis and idealized tests[J]. Advances in Atmospheric Sciences, 2014, 31(3): 693-704.
[17] Chen Dehui, Shen Xueshun. Recent Progress on GRAPES Research and Application[J]. Journal of Applied Meteorological Science, 2006,17(06): 773-777. (in Chinese) 陈德辉, 沈学顺. 新一代数值预报系统 GRAPES 研究进展[J]. 应用气象学报, 2006, 17(6): 773-777.
[18] Qian J H, Semazzi F H M, Scroggs J S. A global nonhydrostatic semi-Lagrangian atmospheric model with orography[J]. Monthly weather review, 1998, 126(3): 747-771.
[19] Robert A. A semi-Lagrangian and semi-implicit numerical integration scheme for the primitive meteorological equations[J]. Journal of the Meteorological Society of Japan. Ser. II, 1982, 60(1): 319-325.
[20] Song Junqiang, Wu Xiangjun, Zhang Lilun, et al. A Study of the Two Helmholtz Solvers in the A Study of the Two Helmholtz Solvers in the GRAPES Model Using GCR andGMRES[J]. Computer Engineering and Science, 2011, 33(11): 65-70. (in Chinese) 宋君强，伍湘君，张理论，等. GRAPES模式中Helmhothz 方程两种求解方法的对比研究[J]. 计算机工程与科学, 2011, 33(11): 65-70.
[21] Xue Jishan, Chen Dehui. Scientific Design and Application of Numerical Prediction System GRAPES [M]. Science Press, 2008. (in Chinese) 薛纪善，陈德辉. 数值预报系统GRAPES的科学设计与应用[M]. 科学出版社, 2008.
[22] Saad Y. Iterative methods for sparse linear systems[M]. Society for Industrial and Applied Mathematics, 2003.
[23] Kreutzer M, Hager G, Wellein G, et al. A unified sparse matrix data format for efficient general sparse matrix-vector multiplication on modern processors with wide SIMD units[J]. SIAM Journal on Scientific Computing, 2014, 36(5): C401-C423.
[24] Zardoshti P, Khunjush F, Sarbazi-Azad H. Adaptive sparse matrix representation for efficient matrix–vector multiplication[J]. The Journal of Supercomputing, 2016, 72(9): 3366-3386.
[25] Chunduri S, Parker S, Balaji P, et al. Characterization of MPI usage on a production supercomputer[C]//SC18: International Conference for High Performance Computing, Networking, Storage and Analysis. IEEE, 2018: 386-400.
[26] Panda D K, Subramoni H, Chu C, et al. The MVAPICH project: Transforming research into high-performance MPI library for HPC community[J]. Journal of Computational Science, 2020: 101208.