E级高性能计算机的维护故障诊断系统研究

doi:10.19678/j.issn.1000-3428.0065704

摘要/Abstract

摘要： E级计算机系统规模巨大，使得故障异常总量随之增多，导致诊断发现的难度增加，因此，迫切需要一套更加准确高效的实时维护故障诊断系统，对硬件系统进行全面的异常及故障信息实时检测、故障诊断及故障预测。传统故障诊断系统在面对数万节点规模的诊断时存在执行效率低、异常检测误报率高的问题，异常检测及故障诊断的覆盖率不足。对异常及故障检测、故障诊断与故障预测相关技术进行研究，分析技术原理及适用性，并结合E级高性能计算机实际工程需求，设计一套满足数E级高性能计算机需求的维护故障诊断系统。基于维护系统的结构组成设计可扩展的边缘诊断架构，将高性能计算机系统知识、专家知识与数理统计、机器学习相融合给出故障检测、诊断及预测算法，并针对专用场景建立预测模型。实验结果表明，该系统具有较好的可扩展性，能在10 s内完成对十万个节点规模系统的故障诊断，与传统故障诊断系统相比，异常检测某特定指标误报率从3.3%降低到几乎为0，硬件故障检测覆盖率从90.2%提升至96%以上，硬件故障诊断覆盖率从71%提升至约94%，能较准确地预测多个重要应用场景下的故障。

关键词: 高性能计算, 维护系统, 异常检测, 故障诊断, 故障预测

Abstract: E-class computer systems typically have huge scales.Consequently, the total number of abnormal faults is bound to increase, resulting in difficulty in fault diagnosis.Thus, there is an urgent need for the development of a more accurate and efficient real-time maintenance fault diagnosis system that is able to perform comprehensive real-time detection, fault diagnosis, and fault prediction of hardware systems using abnormal and fault information. The traditional fault diagnosis system faces the problems of low execution efficiency and high false-positive rate of abnormal detection in the face of tens of thousands of nodes.Additionally, the coverage rate of abnormal detection and fault diagnosis is insufficient.In this study, abnormal and fault detection, fault diagnosis, and fault prediction of related technologies are evaluated, the principle and applicability of the main methods are analyzed, and a set of maintenance fault diagnosis systems that can meet the needs of E-class high-performance computers are designed in combination with the actual engineering requirements of E-class high-performance computers.Moreover, a scalable edge diagnosis architecture is designed based on the structural composition of the maintenance system, and the high-performance computer system knowledge and expert knowledge are integrated using mathematical statistics and machine learning to design fault detection, diagnosis, and prediction algorithms.Finally, prediction models are established for special scenarios.The experimental results show that the system has good scalability compared with the traditional fault diagnosis system, and can complete the fault diagnosis of 100 000 nodes within 10 s. Additionally, the false positive rate of the specific indicator used in anomaly detection is reduced from 3.3% to almost zero, the hardware fault detection coverage rate is increased from 90.2% to more than 96%, the hardware fault diagnosis coverage is increased from 71% to 94%, and the fault prediction can accurately predict the faults in several important application scenarios.

Key words: high-performance computing, maintenance system, anomaly detection, fault diagnosis, fault prediction

中图分类号:

TP302.1

建澜涛, 任秀江, 张祯, 石嵩, 黄益明, 张春林. E级高性能计算机的维护故障诊断系统研究[J]. 计算机工程, 2022, 48(12): 24-37.

JIAN Lantao, REN Xiujiang, ZHANG Zhen, SHI Song, HUANG Yiming, ZHANG Chunlin. Research on Maintenance Fault Diagnosis System for E-class High-Performance Computer[J]. Computer Engineering, 2022, 48(12): 24-37.

http://www.ecice06.com/CN/Y2022/V48/I12/24

图/表 16

20230112182228

20230112182233

20230112182237

20230112182240

20230112182244

20230112182248

20230112182251

20230112182255

20230112182258

20230112182302

20230112182306

20230112182310

20230112182313

20230112182316

20230112182320

20230112182324

参考文献

[1] LIU Y, LIU X, LI F, et al.Closing the "quantum supremacy" gap:achieving real-time simulation of a random quantum circuit using a new sunway supercomputer[EB/OL].[2022-08-01].https://arxiv.org/abs/2110.14502.
[2] LIU X, GUO C, LIU Y, et al.Redefining the quantum supremacy baseline with a new generation sunway supercomputer[EB/OL].[2022-08-01].https://arxiv.org/abs/2111.01066.
[3] PHILP I R.Software failures and the road to a petaflop machine[C]//Proceedings of the 11th International Symposium on High Performance Computer Architecture.Washington D.C., USA:IEEE Computer Society, 2005:235-246.
[4] 胡维, 蒋艳凰, 刘光明, 等.E级超级计算机故障预测的数据采集方法[J].国防科技大学学报, 2016, 38(1):93-100. HU W, JIANG Y H, LIU G M, et al.Data collection for failure prediction toward exascale supercomputers[J].Journal of National University of Defense Technology, 2016, 38(1):93-100.(in Chinese)
[5] JOHNSON T, KWOK I, NG R.Fast computation of 2-dimensional depth contours[C]//Proceedings of the 7th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York, USA:ACM Press, 1998:224-228.
[6] HE Z Y, XU X F, DENG S C.Discovering cluster-based local outliers[J].Pattern Recognition Letters, 2003, 24(9/10):1641-1650.
[7] ARNING A, AGRAWAL R, RAGARAN P.A linear method for deviation detection in large database[C]//Proceedings of the 2nd IEEE International Conference on Knowledge Discovery in Databases and Data Mining.Washington D.C., USA:IEEE Press, 1996:164-169.
[8] JAGADISH H V, NICK K, MUTHUKRISHNAN S.Mining deviants in a time series database[C]//Proceedings of the 25th IEEE International Conference on Very Large Data Bases.Washington D.C., USA:IEEE Press, 1999:7-10.
[9] KNORR E M, NG R T, TUCAKOV V.Distance-based outliers:algorithms and applications[J].The VLDB Journal, 2000, 8(3/4):237-253.
[10] BREUNIG M M, KRIEGEL H P, NG R T, et al.LOF:identifying density-based local outliers[C]//Proceedings of SIGMODʼ00.New York, USA:ACM Press, 2000:93-104.
[11] 赵串串, 游进国, 李晓武.用于下一项推荐的序列感知深度网络[J].小型微型计算机系统, 2020, 41(7):1389-1394. ZHAO C C, YOU J G, LI X W.Sequence-aware deep network for the next-item recommendation[J].Journal of Chinese Computer Systems, 2020, 41(7):1389-1394.(in Chinese)
[12] 杨瑞朋, 屈丹, 朱少卫, 等.基于改进时间卷积网络的日志序列异常检测[J].计算机工程, 2020, 46(8):50-57. YANG R P, QU D, ZHU S W, et al.Anomaly detection for log sequence based on improved temporal convolutional network[J].Computer Engineering, 2020, 46(8):50-57.(in Chinese)
[13] 胡姣姣, 王晓峰, 张萌, 等.基于深度学习的时间序列数据异常检测方法[J].信息与控制, 2019, 48(1):1-8. HU J J, WANG X F, ZHANG M, et al.Time-series data anomaly detection method based on deep learning[J].Information and Control, 2019, 48(1):1-8.(in Chinese)
[14] VENKATASUBRAMANIAN V, RENGASWAMY R, KAVURI S N, et al.A review of process fault detection and diagnosis:part III:process history based methods[J].Computers & Chemical Engineering, 2003, 27(3):327-346.
[15] VENKATASUBRAMANIAN V, RENGASWAMY R, KAVURI S N.A review of process fault detection and diagnosis:part II:qualitative models and search strategies[J].Computers & Chemical Engineering, 2003, 27(3):313-326.
[16] LIU Y J, MENG Q H, ZENG M, et al.Fault diagnosis method based on probability extended SDG and fault index[C]//Proceedings of the 12th World Congress on Intelligent Control and Automation.Washington D.C., USA:IEEE Press, 2016:2868-2873.
[17] SHAKERI M, RAGHAVAN V, PATTIPATI K R, et al.Sequential testing algorithms for multiple fault diagnosis[J].IEEE Transactions on Systems, Man, and Cybernetics-Part A:Systems and Humans, 2000, 30(1):1-14.
[18] GE N, NAKAJIMA S, PANTEL M.Online diagnosis of accidental faults for real-time embedded systems using a hidden Markov model[J].Transactions of the Society for Modeling and Simulation International, 2015, 91(10):851-868.
[19] TYAGI S K, PANDEY D, TYAGI R.Fuzzy set theoretic approach to fault tree analysis[J].International Journal of Engineering, Science and Technology, 2010, 2(5):276-283.
[20] 周志杰, 曹友, 胡昌华, 等.基于规则的建模方法的可解释性及其发展[J].自动化学报, 2021, 47(6):1201-1216. ZHOU Z J, CAO Y, HU C H, et al.The interpretability of rule-based modeling approach and its development[J].Acta Automatica Sinica, 2021, 47(6):1201-1216.(in Chinese)
[21] GAO T R, YU D, YUE D F.Probabilistic neural network based on adaptive error correction model and its application in failure diagnosis[J].Computer Integrated Manufacturing Systems, 2013, 19(11):2824-2833.
[22] 吕瑞, 孙林夫.基于多源信息融合故障树与模糊Petri网的复杂系统故障诊断方法[J].计算机集成制造系统, 2017, 23(8):1817-1831. LÜ R, SUN L F.Fault diagnosis method of complex system based on multi-source information fusion fault tree and fuzzy Petri net[J].Computer Integrated Manufacturing Systems, 2017, 23(8):1817-1831.(in Chinese)
[23] 郑玮, 杨学猛, 葛宁.基于专家系统和神经网络的卫星故障诊断系统设计与实现[J].燕山大学学报, 2016, 40(1):74-80. ZHENG W, YANG X M, GE N.Design and implementation of fault diagnosis system for satellite based on expert system and neural network[J].Journal of Yanshan University, 2016, 40(1):74-80.(in Chinese)
[24] 宋龙龙, 王太勇, 宋晓文, 等.基于Petri网建模与FTA的动车组受电弓故障诊断[J].仪器仪表学报, 2014, 35(9):1990-1997. SONG L L, WANG T Y, SONG X W, et al.Fault diagnosis of pantograph type current collector of CRH2 electric multiple units based on Petri net modeling and fault tree analysis[J].Chinese Journal of Scientific Instrument, 2014, 35(9):1990-1997.(in Chinese)
[25] 张燕, 佘维, 李平.基于Petri网和故障树的电力系统故障诊断模型[J].计算机测量与控制, 2015, 23(8):2626-2628, 2632. ZHANG Y, SHE W, LI P.Fault diagnosis model of electric power systems based on petri nets and fault tree analysis[J].Computer Measurement & Control, 2015, 23(8):2626-2628, 2632.(in Chinese)
[26] 谢光强, 郭小全, 李杨, 等.微型嵌入式系统故障诊断方法综述[J].计算机应用研究, 2019, 36(5):1281-1285, 1292. XIE G Q, GUO X Q, LI Y, et al.Review of fault diagnosis methods for micro-embedded system[J].Application Research of Computers, 2019, 36(5):1281-1285, 1292.(in Chinese)
[27] 袁远, 李世杰, 邢建英, 等.E级高性能计算机系统中监控分系统的挑战与设计[J].计算机工程与科学, 2021, 43(8):1366-1375. YUAN Y, LI S J, XING J Y, et al.Monitoring subsystem for exascale HPC systems:challenges and design[J].Computer Engineering & Science, 2021, 43(8):1366-1375.(in Chinese)
[28] YAO L Y, CHU Z X, LI S, et al.A survey on causal inference[J].ACM Transactions on Knowledge Discovery From Data, 2021, 74(1):46.
[29] ATHEY S, IMBENS G.Machine learning methods for estimating heterogeneous causal effects[J].Statistics, 2015, 5:1-26.
[30] CHIKAHARA Y, FUJINO A.Causal inference in time series via supervised learning[C]//Proceedings of the 27th IEEE International Joint Conference on Artificial Intelligence.Washington D.C., USA:IEEE Press, 2018:2042-2048.
[31] EGAMI N, IMAI K.Causal interaction in factorial experiments:application to conjoint analysis[J].Journal of the American Statistical Association, 2019, 114(526):529-540.
[32] GUO R C, LU C, LI JD, et al.A survey of learning causality with data:Problems and methods[J].ACM Computing Surveys, 2020, 53(4):37-75.
[33] HAHN P R, MURRAY J S, CARVALHO C M.Bayesian regression tree models for causal inference:regularization, confounding, and heterogeneous effects (with discussion)[J].Bayesian Analysis, 2020, 15(3):965-1056.
[34] IMBENS G.Potential outcome and directed acyclic graph approaches to causality:relevance for empirical practice in economics[J].Journal of Economic Literature, 2020, 58(4):1129-1179.
[35] FU X, REN R, ZHAN J, et al.LogMaster:mining event correlations in logs of large-scale cluster systems[C]//Proceedings of IEEE International Symposium on Reliable Distributed Systems.Washington D.C., USA:IEEE Computer Society, 2012:71-80.
[36] 韩东, 杨震, 许葆华.基于数据驱动的故障预测模型框架研究[J].计算机工程与设计, 2013, 34(3):1054-1058. HAN D, YANG Z, XU B H.Research on fault prognostics model and frame based on data-driven method[J].Computer Engineering and Design, 2013, 34(3):1054-1058.(in Chinese)
[37] LIANG Y, ZHANG Y Y, XIONG H, et al.Failure prediction in IBM BlueGene/L event logs[C]//Proceedings of the 7th IEEE International Conference on Data Mining.Washington D.C., USA:IEEE Press, 2007:583-588.
[38] 胡维.面向超级计算机的故障预测和容错关键技术研究[D].长沙:国防科技大学, 2017. HU W.Research on failure prediction and fault-tolerance technology for supercomputer[D].Changsha:National University of Defense Technology, 2017.(in Chinese)
[39] 陆兴海, 彭华盛.运维数据治理——构筑智能运维的基石[M].北京:机械工业出版社, 2022. LU X H, PENG H S.Operation and maintenance data governance:the cornerstone of building intelligent operation and maintenance[M].Beijing:China Machine Press, 2022.(in Chinese)

选择文件类型/文献管理软件名称

选择包含的内容