基于异构Flink集群的节点优先级调度策略

doi:10.19678/j.issn.1000-3428.0059944

计算机工程 ›› 2022, Vol. 48 ›› Issue (3): 197-203. doi: 10.19678/j.issn.1000-3428.0059944

基于异构Flink集群的节点优先级调度策略

汪文豪¹, 史雪荣²

1. 南京工业大学计算机科学与技术学院, 南京 211816;
2. 盐城师范学院数学与统计学院, 江苏盐城 224002

收稿日期:2020-11-09 修回日期:2021-02-20 发布日期:2022-03-11
作者简介:汪文豪(1997-),男,硕士研究生,主研方向为分布式应用;史雪荣,教授、博士。
基金资助:
国家自然科学基金（11872327）；江苏省高等学校自然科学研究项目（20KJA190001）。

Node Priority Scheduling Strategy Based on Heterogeneous Flink Cluster

WANG Wenhao¹, SHI Xuerong²

1. College of Computer Science and Technology, Nanjing Tech University, Nanjing 211816, China;
2. College of Mathematics and Statistics, Yancheng Teachers University, Yancheng, Jiangsu 224002, China

Received:2020-11-09 Revised:2021-02-20 Published:2022-03-11

摘要/Abstract

摘要： Flink流处理系统默认的任务调度策略在一定程度上忽略了集群异构和节点可用资源，导致集群整体负载不均衡。研究分布式节点的实时性能和集群作业环境，根据实际作业环境的异构分布情况，设计结合异构Flink集群的节点优先级调整方法，以基于Ganglia可扩展分布式集群资源监控系统的集群信息为依据，动态调整适应当前作业环境的节点优先级指数。基于此提出Flink节点动态自适应调度策略，通过实时监测节点的异构状况，并在任务执行过程中根据实时作业环境更新节点优先级指数，为系统任务找到最佳的执行节点完成任务分配。实验结果表明，相比于Flink默认的任务调度策略，基于节点优先级调整方法的自适应调度策略在WorldCount基准测试中的运行时间约平均减少6%，可使异构Flink集群在保持集群低延迟的同时，节点资源利用率和任务执行效率更高。

关键词: Flink集群, 异构集群, 负载不均衡, 节点优先级, 自适应调度

Abstract: The default task scheduling strategy of the Flink stream processing system ignores the cluster heterogeneity and available resources of nodes to a certain extent, resulting in an unbalanced overall cluster load.This study investigates the real-time performance of distributed nodes and the cluster operation environment and designs a node priority-adjustment method based on heterogeneous Flink clusters according to the heterogeneous distribution problem of the actual operation environment.The method dynamically adjusts the node priority index that adapts to the current operating environment based on the cluster information of the Ganglia scalable distributed cluster resource-monitoring system.Based on this, a dynamic adaptive scheduling strategy for link nodes is proposed.By monitoring the heterogeneous status of nodes in real time and updating the node priority index according to the real-time working environment during the task execution process, the best execution node for the system task to complete the task assignment can be found.The experimental results show that compared with Flink's default task scheduling strategy, the adaptive scheduling strategy based on the node priority adjustment method reduces the running time of the WorldCount benchmark by approximately 6% on average.This enables the heterogeneous Flink cluster to maintain low cluster latency while maintaining higher node resource utilization and task execution efficiency.

Key words: Flink cluster, heterogeneous cluster, load unbalancing, node priority, adaptive scheduling

中图分类号:

TP393

汪文豪, 史雪荣. 基于异构Flink集群的节点优先级调度策略[J]. 计算机工程, 2022, 48(3): 197-203.

WANG Wenhao, SHI Xuerong. Node Priority Scheduling Strategy Based on Heterogeneous Flink Cluster[J]. Computer Engineering, 2022, 48(3): 197-203.

http://www.ecice06.com/CN/Y2022/V48/I3/197

图/表 8

20220331202523

20220331202526

20220331202529

20220331202532

20220331202535

20220331202538

20220331202541

20220331202544

参考文献

[1] 孙大为, 张广艳, 郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报, 2014, 25(4):839-862. SUN D W, ZHANG G Y, ZHENG W M.Big data flow computing:key technologies and system examples[J].Journal of Software, 2014, 25(4):839-862.(in Chinese)
[2] SHAHVERDI E, AWAD A, SAKR S.Big stream processing systems:an experimental evaluation[C]//Proceedings of the 35th International Conference on Data Engineering Workshops.Washington D.C., USA:IEEE Press, 2019:53-60.
[3] QIN X, JIANG H.A dynamic and reliability-driven scheduling algorithm for parallel real-time jobs executing on heterogeneous clusters[J].Journal of Parallel and Distributed Computing, 2005, 65(8):885-900.
[4] ZHU X M, HE C, LI K L, et al.Adaptive energy-efficient scheduling for real-time tasks on DVS-enabled heterogeneous clusters[J].Journal of Parallel and Distributed Computing, 2012, 72(6):751-763.
[5] WU Y, WANG Z R, RUAN Q, et al.Node scheduling:a blockchain-based node selection approach on sapiens chain[C]//Proceedings of 2019 IEEE International Conference on Big Data and Smart Computing.Washington D.C., USA:IEEE Press, 2019:1-7.
[6] ALI-ELDIN A, TORDSSON J, ELMROTH E.An adaptive hybrid elasticity controller for cloud infrastructures[C]//Proceedings of 2012 IEEE Network Operations and Management Symposium.Washington D.C., USA:IEEE Press, 2012:204-212.
[7] TRAN G P C, CHEN Y, KANG D I, et al.Automated demand-based vertical elasticity for heterogeneous real-time workloads[C]//Proceedings of the 9th International Conference on Cloud Computing.Washington D.C., USA:IEEE Press, 2016:831-834.
[8] JAYAKUMAR V K, LEE J, KIM I K, et al.A self-optimized generic workload prediction framework for cloud computing[C]//Proceedings of 2020 IEEE International Parallel and Distributed Processing Symposium.Washington D.C., USA:IEEE Press, 2020:779-788.
[9] MICHELLE Y, GAREGRAT N, MOHAN S.Towards a resource aware scheduler in Hadoop[C]//Proceedings of the 7th IEEE International Conference on Web Services.Washington D.C., USA:IEEE Press, 2009:102-109.
[10] CARUANA G, LI M Z, QI M, et al.gSched:a resource aware Hadoop scheduler for heterogeneous cloud computing environments[J].Concurrency and Computation:Practice & Experience, 2017, 29(20):1-7.
[11] NAIK N S, NEGI A, BAPU T B R, et al.A data locality based scheduler to enhance MapReduce performance in heterogeneous environments[J].Future Generation Computer Systems, 2019, 90:423-434.
[12] XU X L, CAO L L, WANG X H.Adaptive task scheduling strategy based on dynamic workload adjustment for heterogeneous Hadoop clusters[J].IEEE Systems Journal, 2017, 10(2):471-482.
[13] 廉华, 刘瑜.基于YARN资源调度器的MapReduce作业数调节方法[J].计算机系统应用, 2020, 29(3):218-222. LIAN H, LIU Y.Number adjustment method of MapReduce jobs based on YARN resource scheduler[J].Computer Systems & Applications, 2020, 29(3):218-222.(in Chinese)
[14] 杨志伟, 郑烇, 王嵩, 等.异构Spark集群下自适应任务调度策略[J].计算机工程, 2016, 42(1):31-35, 40. YANG Z W, ZHENG Q, WANG S, et al.Adaptive task scheduling strategy for heterogeneous spark cluster[J].Computer Engineering, 2016, 42(1):31-35, 40.(in Chinese)
[15] 胡亚红, 盛夏, 毛家发.资源不均衡Spark环境任务调度优化算法研究[J].计算机工程与科学, 2020, 42(2):203-209. HU Y H, SHENG X, MAO J F.Task scheduling optimization in Spark environment with unbalanced resources[J].Computer Engineering & Science, 2020, 42(2):203-209.(in Chinese)
[16] 郝志峰, 黄泽林, 蔡瑞初, 等.基于YARN的分布式资源动态调度和协同分配系统[J].计算机工程, 2021, 47(2):226-232. HAO Z F, HUANG Z L, CAI R C, et al.Dynamic resource scheduling and collaborative allocation system based on YARN[J].Computer Engineering, 2021, 47(2):226-232.(in Chinese)
[17] 何贞贞, 于炯, 李梓杨, 等.基于Flink的任务调度策略[J].计算机工程与设计, 2020, 41(5):1280-1287. HE Z Z, YU J, LI Z Y, et al.Task scheduling strategy based on Flink environment[J].Computer Engineering and Design, 2020, 41(5):1280-1287.(in Chinese)
[18] 庆骁.面向FLINK流处理框架的容错策略优化研究[D].哈尔滨:哈尔滨工业大学, 2019. QING X.Research on fault-tolerant strategy optimization for FLINK stream processing framework[D].Harbin:Harbin Institute of Technology, 2019.(in Chinese)
[19] 赵娟, 程国钟.基于Hadoop、Storm、Samza、Spark及Flink大数据处理框架的比较研究[J].信息系统工程, 2017(6):117, 119. ZHAO J, CHEN G Z.Comparison of big data processing frameworks based on Hadoop, Storm, Samza, Spark and Flink[J].China CIO News, 2017(6):117, 119.(in Chinese)
[20] TOLIOPOULOS T, GOUNARIS A.Adaptive distributed partitioning in Apache Flink[C]//Proceedings of the 36th International Conference on Data Engineering Workshops.Washington D.C., USA:IEEE Press, 2020:127-132.
[21] Ganglia[EB/OL].[2020-10-19].http://ganglia.info/.
[22] AJILA T, MAJUMDAR S.Data driven priority scheduling on a spark streaming system[C]//Proceedings of the 19th IEEE/ACM International Symposium on Cluster, Cloud and Grid Computing.Washington D.C., USA:IEEE Press, 2019:561-568.
[23] KAMSKY A.Adapting TPC-C benchmark to measure performance of multi-document transactions in MongoDB[J].Proceedings of the VLDB Endowment, 2019, 12(12):2254-2262.

选择文件类型/文献管理软件名称

选择包含的内容

基于异构Flink集群的节点优先级调度策略

Node Priority Scheduling Strategy Based on Heterogeneous Flink Cluster

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献

相关文章 5

编辑推荐

Metrics

本文评价

[1]	许武,梁军,李威,徐鹏飞,徐圣瑞,张福贵. 异构计算平台激光雷达算法优化研究[J]. 计算机工程, 2018, 44(7): 1-7.
[2]	杨志伟,郑烇,王嵩,杨坚,周乐乐. 异构Spark集群下自适应任务调度策略[J]. 计算机工程, 2016, 42(1): 31-35,40.
[3]	丁晓波,马中,戴新发. 面向响应延迟的虚拟机动态时间片调度算法[J]. 计算机工程, 2015, 41(7): 11-16,24.
[4]	刘亭;王占林;裘丽华. 机载公共设备管理系统中的自适应调度算法[J]. 计算机工程, 2008, 34(9): 240-242.
[5]	张昊;廖建新;朱晓民. 增强型动态反馈随机分发负载均衡算法[J]. 计算机工程, 2007, 33(04): 97-99.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于异构Flink集群的节点优先级调度策略

Node Priority Scheduling Strategy Based on Heterogeneous Flink Cluster

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献

相关文章 5

编辑推荐

Metrics

本文评价