摘要： Storm分布式实时计算系统采用轮询调度算法达到任务均衡分配的目的,但Storm默认调度算法未考虑计算节点间的网络通信开销,可能造成topology处理时延高、集群吞吐量低等问题。为此,引入Storm topology热边的概念,通过减少网络传输的tuple数量,将热边关联的源executor和目标executor调度到同一工作节点执行,提升Storm集群的计算性能。实验结果表明,该调度算法能够有效降低Storm处理时延及集群节点间的通信量,提高系统吞吐量,并且有利于集群的负载均衡。
关键词:
实时计算,
拓扑结构热边,
调度算法,
负载均衡,
网络通信
Abstract: Storm is an open source distributed real-time computation system.It takes round-robin scheduling algorithm to achieve task distribution balancing,but the Storm default scheduling algorithm disregards inter-node network communication overhead,which may cause the problem of high processing latency and low throughput.To solve these problems,the concept of Storm topology hot-edge is proposed.The proposed algorithm can reduce the number of tuple which is transmitted through network by scheduling the hot-edge connected source exeactor and target executor to the same cluster node,which in turn promotes calculation performance of Storm cluster.Experimental results show that the proposed scheduling algorithm can reduce the Storm processing latency and cluster inter-node traffic,and improve system throughput,which is conducive to cluster load balancing.
Key words:
real-time computing,
topology hot-edge,
scheduling algorithm,
load balancing,
network communication
中图分类号: