Dynamic Scheduling and Collaborative Allocation System for Distributed Resource Based on YARN

doi:10.19678/j.issn.1000-3428.0057292

Computer Engineering ›› 2021, Vol. 47 ›› Issue (2): 226-232. doi: 10.19678/j.issn.1000-3428.0057292

• Computer Architecture and Software Technology • Previous Articles Next Articles

Dynamic Scheduling and Collaborative Allocation System for Distributed Resource Based on YARN

HAO Zhifeng^1,2, HUANG Zelin¹, CAI Ruichu¹, FU Zhengjia¹, WEN Wen¹, TANG Kailin¹

1. College of Computer, Guangdong University of Technology, Guangzhou 510006, China;
2. College of Mathematics and Big Data, Foshan University, Foshan, Guangdong 528000, China

Received:2020-01-27 Revised:2020-03-26 Online:2021-02-15 Published:2020-03-31

基于YARN的分布式资源动态调度与协同分配系统

郝志峰^1,2, 黄泽林¹, 蔡瑞初¹, 傅正佳¹, 温雯¹, 唐凯麟¹

1. 广东工业大学计算机学院, 广州 510006;
2. 佛山科学技术学院数学与大数据学院, 广东佛山 528000

作者简介:郝志峰(1968-),男,教授、博士,主研方向为机器学习、人工智能;黄泽林(通信作者),硕士研究生;蔡瑞初,教授、博士;傅正佳,博士;温雯,副教授、博士;唐凯麟,硕士。
基金资助:
国家自然科学基金（61876043）；NSFC-广东联合基金（U1501254）；广东省自然科学基金（2014A030306004，2014A030308008）；广东特支计划（2015TQ01X140）；广州市珠江科技新星专项（201610010101）；广州市科技计划项目（201902010058）。

Abstract

Abstract: Storm on YARN is one of the mainstream distributed resource scheduling frameworks.However,it requires manual intervention,so it fails to adjust the system resources according to resource availability in real time.To address the problem,this paper proposes a distributed resource scheduling and collaborative allocation model based on YARN on the Storm platform,where the loads of the system is estimated based on the real-time delay of the streaming data process.The model is based on a two-layer scheduling model,including the system layer and the task layer.The system layer predicts the resource allocation through real-time monitoring of streaming data processing loads,and the ZooKeeper-based task layer uses YARN's efficient management of cluster resources to dynamically manage them. Experimental results show that the proposed system can dynamically adjust the cluster resources distribution in real time to reduce the system delay.

Key words: distributed cluster, dynamic scheduling, collaborative allocation, stream data processing, resource allocation

摘要： Storm on YARN是目前主流的分布式资源调度框架，但其存在需要人工干预和无法根据资源可用性实时调整系统资源的不足。根据流数据处理的实时延迟计算系统负载情况，在Storm平台上基于YARN设计分布式资源调度和协同分配系统。建立包含系统层和任务层的双层调度模型，系统层通过对流数据处理负载的实时监测进行资源分配预测，任务层利用ZooKeeper和YARN对集群资源的高效管理能力进行动态资源管理。实验结果表明，该系统可以实时调整集群资源分布，有效减小系统延迟。

关键词: 分布式集群, 动态调度, 协同分配, 流数据处理, 资源分配

CLC Number:

TP315

HAO Zhifeng, HUANG Zelin, CAI Ruichu, FU Zhengjia, WEN Wen, TANG Kailin. Dynamic Scheduling and Collaborative Allocation System for Distributed Resource Based on YARN[J]. Computer Engineering, 2021, 47(2): 226-232.

郝志峰, 黄泽林, 蔡瑞初, 傅正佳, 温雯, 唐凯麟. 基于YARN的分布式资源动态调度与协同分配系统[J]. 计算机工程, 2021, 47(2): 226-232.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0057292

http://www.ecice06.com/EN/Y2021/V47/I2/226

Figures/Tables 9

References 21

[1]	LIU Pengcheng,CHEN Rong.Cloud computing-oriented live migration framework for virtual machine[J].Computer Engineering,2010,36(5):37-39.(in Chinese)刘鹏程,陈榕.面向云计算的虚拟机动态迁移框架[J].计算机工程,2010,36(5):37-39.
[2]	COULOURIS G,DOLLIMORE J,KINDBER T,et al.Distributed systems:concepts and design[M].Translated by JIN Beihong,MA Yinglong.5th ed.Beijing:Machinery Industry Press,2013.(in Chinese)COULOURIS G,DOLLIMORE J,KINDBER T,等.分布式系统:概念与设计[M].金蓓弘,马应龙,译.5版.北京:机械工业出版社,2013.
[3]	SUN Dawei,ZHANG Guangyan,ZHENG Weimin.Big data stream computing:technologies and instances[J].Journal of Software,2014,25(4):839-862.(in Chinese)孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.
[4]	NANDIMATH J,BANERJEE E,PATIL A,et al.Big data analysis using Apache Hadoop[C]//Proceedings of IEEE International Conference on Information Reuse and Integration.Washington D.C.,USA:IEEE Press,2013:700-703.
[5]	HAN Zhijie,ZHANG Yujie.Spark:a big data processing platform based on memory computing[C]//Proceedings of the 7th International Symposium on Parallel Architectures,Algorithms and Programming.Washington D.C.,USA:IEEE Press,2016:172-176.
[6]	TOSHNIWAL A,TANEJA S,SHUKLA A,et al.Storm@Twitter[C]//Proceedings of ACM SIGMOD International Conference on Management of Data.New York,USA:ACM Press,2014:147-156.
[7]	CARBONE P,KATSIFODIMOS A,EWEN S,et al.Apache FlinkTM:stream and batch processing in a single engine[J].Bulletin of the IEEE Computer Society Technical Committee on Data Engineering,2015,36(4):1-10.
[8]	VAVILAPALLI V K,MURTHY A C,DOUGLAS C,et al.Apache Hadoop YARN:yet another resource negotiator[C]//Proceedings of the 4th Annual Symposium on Cloud Computing.New York,USA:ACM Press,2013:1-16.
[9]	HINDMAN B,KONWINSKI A,ZAHARIA M,et al.Mesos:a platform for fine-grained resource sharing in the data center[C]//Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementa-tion.New York,USA:ACM Press,2011:295-308.
[10]	XIAN Yongju,ZHENG Jian,XU Changbiao.Clustering-based user resource allocation scheme for small base station[J].Computer Engineering,2019,45(5):110-115.(in Chinese)鲜永菊,郑健,徐昌彪.基于分簇的小基站用户资源分配方案[J].计算机工程,2019,45(5):110-115.
[11]	LIU Yi,ZHANG Kan.Strategy for workflow task assignment based on load balance and experiential value[J].Computer Engineering,2009,35(21):57-59.(in Chinese)刘怡,张戡.基于负载平衡和经验值的工作流任务分配策略[J].计算机工程,2009,35(21):57-59.
[12]	HUNT P,KONAR M,et al.ZooKeeper:wait-free coordination for Internet-scale systems[EB/OL].[2019-12-20].http://www.usenix.org/events/atc10/tech/full_papers/Hunt.pdf.
[13]	FU T Z J,DING J,MA R T B,et al.DRS:auto-scaling for real-time stream analytics[J].IEEE/ACM Transactions on Networking,2017,25(6):3338-3352.
[14]	Github Inc.Yahoo/Storm on YARN[EB/OL].[2019-12-20]. https://github.com/yahoo/storm-yarn.
[15]	Apache Corporation.Flink on YARN[EB/OL].[2019-12-20]. https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/yarn_setup.html#flink-yarn-session.
[16]	Github Inc.Storm on Mesos[EB/OL].[2019-12-20].https://github.com/mesos/storm.
[17]	Apache Corporation.Flink on Mesos[EB/OL].[2019-12-20].https://ci.apache.org/projects/flink/flink-docs-release-1.3/setup/mesos.html.
[18]	Github Inc.Storm Wiki[EB/OL].[2019-12-20].https://github.com/apache/storm.
[19]	CHEN Feng.Distributed cluster storage and its advantages[J].Radio and Television Information,2015(2):31-33.(in Chinese)陈峰.分布式集群存储及其优势浅析[J].广播电视信息,2015(2):31-33.
[20]	YAN Xiaofeng,ZHANG Dexin.Big data research[J].Computer Technology and Development,2013,4:168-172.(in Chinese)严霄凤,张德馨.大数据研究[J].计算机技术与发展,2013,4:168-172.
[21]	SIDDIQA A,KARIM A,GANI A.Overview of big data storage technology[J].Frontiers of Information Technology & Electronic Engineering,2017,18(8):21-24.

Please choose a citation manager

Content to export