数据驱动的自适应容错技术研究

doi:10.19678/j.issn.1000-3428.0050299

计算机工程 ›› 2018, Vol. 44 ›› Issue (12): 46-55. doi: 10.19678/j.issn.1000-3428.0050299

数据驱动的自适应容错技术研究

刘睿涛¹,陈左宁²

1.数学工程与先进计算国家重点实验室,江苏无锡 214215; 2.国家并行计算机工程技术研究中心,北京 100190

收稿日期:2018-01-26 出版日期:2018-12-15 发布日期:2018-12-15
作者简介:刘睿涛(1977—),男,博士研究生,主研方向为高性能计算、并行损伤系统、容错技术;陈左宁,博士生导师、中国工程院院士。
基金资助:
国家重点研发计划(2016YFB0200502)

Study of Data-driven Self-adaptive Fault-tolerant Technology

LIU Ruitao ¹,CHEN Zuoning ²

1.State Key Laboratory of Mathematical Engineering and Advanced Computing,Wuxi,Jiangsu 214215,China; 2.National Research Center of Parallel Computer Engineering and Technology,Beijing 100190,China

Received:2018-01-26 Online:2018-12-15 Published:2018-12-15

摘要/Abstract

摘要：

基于系统故障数据建立层次化失效模型,有助于优化检查点,提升系统可用性水平,应对未来E级计算的可靠性挑战。以太湖之光系统为研究主体,介绍故障采集、分类与处理机制,基于实际故障数据建立细粒度失效分布模型及应用级失效模型的多层失效模型,量化应用运行环境的可靠性。以该模型为基础,分析自适应的检查点容错优化模型,为检查点优化提供理论与工程依据。以太湖之光系统为例进行检查点的容错优化分析,结果表明,数据驱动的自适应容错模型可有效降低系统检查点开销。

关键词: 超级计算机, 失效模型, 数据驱动, 自适应, 容错技术

Abstract:

Establishing a hierarchical failure model based on system fault data helps to optimize checkpoints,and improves system availability,and addresses the reliability challenges of future Exascale computing.Taking Sunway TaihuLight as the research subject,the fault collection,classification and processing mechanism are introduced.Based on the actual fault data,a multi-layer failure model including fine-grained failure distribution model and application-level failure model is established to quantify the reliability of the application operating environment.Based on the model,the adaptive checkpoint fault-tolerant optimization model is analyzed to provide theoretical and engineering basis for checkpoint optimization.Taking Sunway TaihuLight as an example,the fault-tolerant optimization analysis result of checkpoints shows that data-driven adaptive fault-tolerant model can effectively reduce system checkpoint overhead.

Key words: supercomputer, failure model, data-driven, self-adaptive, fault-tolerant technology

中图分类号:

TP391

刘睿涛,陈左宁. 数据驱动的自适应容错技术研究[J]. 计算机工程, 2018, 44(12): 46-55.

LIU Ruitao,CHEN Zuoning. Study of Data-driven Self-adaptive Fault-tolerant Technology[J]. Computer Engineering, 2018, 44(12): 46-55.

http://www.ecice06.com/CN/Y2018/V44/I12/46

参考文献

［1］CAPPELLO F.Resilience:one of the main challenges for exascale computing［D］.Champaign,USA:NRIA Illinois Joint-Laboratory on Petascale Computing,2011.
［2］KUSNEZOV D,BINKLEY S,HARROD B,et al.DOE exascale initiative［EB/OL］.［2017-10-21］.https://energy.gov/downloads/doe-exascale-initiative.
［3］KOGGE P,BERGMAN K,BORKAR S et al.Exascale computing study:technology challenges in achieving exascale systems［EB/OL］.［2017-10-21］.http://www.cse.nd.edu/Reports/2008/TR-2008-13.pdf.
［4］SCHROEDER B,GIBSON G A.A large-scale study of failures in high-performance computing systems［J］.IEEE Transactions on Dependable and Secure Computing,2010,7(4):337-350.
［5］LIANG Y,ZHANG Y,JETTE M,et al.BlueGene/L failure analysis and prediction models［C］//Proceedings of the 43rd Annual IEEE International Conference on Dependable Systems and Networks.Washington D.C.,USA:IEEE Press,2006:425-434.
［6］ZHENG Z,YU L,TANG W et al.Co-analysis of RAS log and job log on blue gene/P［C］//Proceedings of the 2011 IEEE International Parallel & Distributed Process-ing Symposium.Washington D.C.,USA:IEEE Press,2011:840-851.
［7］HEIEN E,LAPINE D,KONDO D,et al.Modeling and tolerat-ing heterogeneous failures in large parallel systems［C］//Proceedings of the 2011 International Conference for High Performance Computing,Networking,Storage and Analysis.Washington D.C.,USA:IEEE Press,2011:214-222.
［8］SCHROEDER B,PINHEIRO E,WEBER W.DRAM errors in the wild:a large-scale field study［C］//Proceedings of the 11th International Joint Conference on Measurement and Modeling of Computer Systems.Washington D.C.,USA:IEEE Press,2009:193-204.
［9］PINHEIRO E,WEBER W,BARROSO L A.Failure trends in a large disk drive population［C］//Proceedings of the 5th USENIX Conference on File and Storage Technologies.Washington D.C.,USA:IEEE Press,2007:17-28.
［10］NIE B,TIWARI D,GUPTA S et al.A large-scale study of soft-errors on GPUs in the field［C］//Proceedings of the 2016 IEEE International Symposium on High Performance Computer Architecture.Washington D.C.,USA:IEEE Press,2016:519-530.
［11］ZHENG Z,LAN Z,GUPTA R,et al.A practical failure prediction with location and lead time for blue gene/P［C］//Proceedings of the 2010 International Conference Dependable Systems and Networks Work-shops.Washington D.C.,USA:IEEE Press,2010:125-134.
［12］SAHOO R K,OLINER A J,RISH I,et al.Critical event prediction for proactive management in large-scale computer clusters［C］//Proceedings of the 9th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York,USA:ACM Press,2003:426-435.
［13］GU J,ZHENG Z,LAN Z,et al.Dynamic meta-learning for failure prediction in large-scale systems:a case study［C］//Proceedings of IEEE International Conference on Parallel Processing.Washington D.C.,USA:IEEE Press,2008:154-166.
［14］GAINARU A,CAPPELLO F,SNIR M,et al.Fault prediction under the microscope:a closer look into HPC systems［C］//Proceedings of IEEE International Conference on High Performance Computing,Networking,Storage and Analysis.Washington D.C.,USA:IEEE Press,2012:456-465.
［15］LU X,WANG H Q,ZHOU R J,et al.Autonomic failure prediction based on manifold learning for large-scale distributed systems［J］.Journal of China Universities of Posts and Telecommunications,2010,17(4):116-124.
［16］YOUNG J W.A first order approximation to the optimum checkpoint interval［J］.Communications of the ACM,1994,17(9):530-531.
［17］GUNAWI H S,HAO M,SUMINTO R O,et al.Why does the cloud stop computing?:lessons from hundreds of service outages［C］//Proceedings of the 7th ACM Symposium on Cloud Computing.New York,USA:ACM Press,2016:1-16.
［18］GUNAWI H S,HAO M,LEESATAPORNWONGSE T,et al.What bugs live in the cloud? A study of 3000+ issues in cloud systems［C］//Proceedings of ACM Symposium on Cloud Computing.New York,USA:ACM Press,2014:1-14.
［19］HUANG P,GUO C,ZHOU L,et al.Gray failure:the achilles’ heel of cloud-scale systems［C］//Proceedings of the 16th Workshop on Hot Topics in Operating Systems.Washington D.C.,USA:IEEE Press,2017:150-155.
［20］ZHENG Z,LAN Z,PARK B H,et al.System log pre-processing to improve failure prediction［C］//Proceedings of IEEE/IFIP International Conference on Dependable Systems and Networks.Washington D.C.,USA:IEEE Press,2009:215-226.
［21］张志华.可靠性理论及工程应用［M］.北京:科学出版社,2012.

[1]	江雨燕, 陶承凤, 李平. 数据增强和自适应自步学习的深度子空间聚类算法[J]. 计算机工程, 2023, 49(8): 96-103, 110.
[2]	张欣怡, 张飞, 郝斌, 高鹭, 任晓颖. 基于改进YOLOv5的口罩佩戴检测算法[J]. 计算机工程, 2023, 49(8): 265-274.
[3]	马娜, 温廷新, 贾旭, 李晓会. 复杂光照条件下自适应的车脸重识别模型[J]. 计算机工程, 2023, 49(8): 275-282, 290.
[4]	汤卫芬, 高翠芳. 极值点自适应加权的动态时间规整算法[J]. 计算机工程, 2023, 49(7): 150-160.
[5]	梅晶, 戴龙宝, 童钊, 邓昕, 王嘉珂. 资源约束下基于Lyapunov优化的自适应卸载算法[J]. 计算机工程, 2023, 49(7): 34-46.
[6]	蔡倩倩, 史旭华. 自适应迁移的分解多目标多任务进化算法[J]. 计算机工程, 2023, 49(7): 55-64.
[7]	顾轶寅, 王鸿奎, 殷海兵. 基于上下文自适应阈值剪枝的快速依赖量化算法[J]. 计算机工程, 2023, 49(7): 143-149.
[8]	王爱玲, 马文臻, 邹自明, 钟佳. 基于领域自适应的卫星工程参数异常检测[J]. 计算机工程, 2023, 49(5): 29-37,47.
[9]	石进, 徐杨, 曹斌. 基于自适应三线性池化网络的细粒度图像分类[J]. 计算机工程, 2023, 49(5): 239-246,254.
[10]	李雪松, 张锲石, 宋呈群, 康宇航, 程俊. 自动驾驶场景下的轨迹预测技术综述[J]. 计算机工程, 2023, 49(5): 1-11.
[11]	王博, 张远, 杨咏蓓. 基于模仿学习的决策树码率自适应算法研究[J]. 计算机工程, 2023, 49(5): 206-214.
[12]	李培育, 张雅丽. 基于改进SRGAN模型的人脸图像超分辨率重建[J]. 计算机工程, 2023, 49(4): 199-205.
[13]	刘晶晶, 黄浩. 引入非局部模块卷积神经网络的基频提取模型[J]. 计算机工程, 2023, 49(3): 128-133,160.
[14]	余嘉昕, 王春媛, 韩华, 高燕. 基于融合代价和优化引导滤波的立体匹配算法[J]. 计算机工程, 2023, 49(3): 257-262,270.
[15]	王国栋, 叶剑, 谢萦, 钱跃良. 基于梯度的自适应阈值结构化剪枝算法[J]. 计算机工程, 2022, 48(9): 113-120.

选择文件类型/文献管理软件名称

选择包含的内容

数据驱动的自适应容错技术研究

Study of Data-driven Self-adaptive Fault-tolerant Technology

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

数据驱动的自适应容错技术研究

Study of Data-driven Self-adaptive Fault-tolerant Technology

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价