摘要： Storm on YARN是目前主流的分布式资源调度框架，但其存在需要人工干预和无法根据资源可用性实时调整系统资源的不足。根据流数据处理的实时延迟计算系统负载情况，在Storm平台上基于YARN设计分布式资源调度和协同分配系统。建立包含系统层和任务层的双层调度模型，系统层通过对流数据处理负载的实时监测进行资源分配预测，任务层利用ZooKeeper和YARN对集群资源的高效管理能力进行动态资源管理。实验结果表明，该系统可以实时调整集群资源分布，有效减小系统延迟。

关键词: 分布式集群, 动态调度, 协同分配, 流数据处理, 资源分配

Abstract: Storm on YARN is one of the mainstream distributed resource scheduling frameworks.However,it requires manual intervention,so it fails to adjust the system resources according to resource availability in real time.To address the problem,this paper proposes a distributed resource scheduling and collaborative allocation model based on YARN on the Storm platform,where the loads of the system is estimated based on the real-time delay of the streaming data process.The model is based on a two-layer scheduling model,including the system layer and the task layer.The system layer predicts the resource allocation through real-time monitoring of streaming data processing loads,and the ZooKeeper-based task layer uses YARN's efficient management of cluster resources to dynamically manage them. Experimental results show that the proposed system can dynamically adjust the cluster resources distribution in real time to reduce the system delay.

Key words: distributed cluster, dynamic scheduling, collaborative allocation, stream data processing, resource allocation

中图分类号: 

• TP315