摘要： 随着分布式深度神经网络训练集群中训练任务数量的增加，通信代价逐渐成为影响训练任务完成时间的重要因素，多任务之间的网络共享问题日益突出。现有研究主要围绕分布式训练通信优化和网络资源调度展开，通过任务粒度调度或单任务通信优化方法降低通信开销。然而，任务粒度调度的方法无法区分任务内不同流的优先级，单任务通信优化方法未充分考虑多任务并发训练场景下通信流之间的竞争关系，这些方法都无法有效解决训练参数同步通信过程中关键通信流延迟引发的掉队者效应，难以实现网络资源的高效协调利用。 针对上述问题，提出了一种基于流粒度优先级与路径选择协同优化的多任务通信流调度方法。首先，对环规约架构下多任务通信流共享网络资源的问题进行数学建模，以所有任务的流优先级配置方案和路径选择方案作为模型的输入，在“交换机同优先级队列内带宽均分”的假设下估计各通信流的吞吐率，并进一步计算各训练任务的通信完成时间，以最小化所有训练任务的整体通信时间为优化目标。由于直接联合求解所有任务的优先级与路径配置具有指数级时间复杂度，难以满足大规模训练场景下的实时调度需求，因此设计了一种结合局部最优求解与运行时反馈机制的近似求解算法，通过二分搜索与回溯搜索相结合的方式降低求解复杂度，同时兼顾求解效率与调度质量。运行过程中，反馈机制能够根据掉队通信流的监测结果动态调整流优先级，从而进一步缓解通信阶段的掉队者效应。 为验证所提方法的有效性，构建了基于容器技术和虚拟交换机的分布式训练仿真平台，对训练节点和网络交换机进行模拟，并设计实现了包含中心调度器、节点代理和任务运行时监测模块的多任务流调度系统。中心调度器负责计算通信流优先级及路径配置，并通过远程过程调用和 OpenFlow 协议分别完成节点流量控制策略和交换机转发规则的动态更新；任务运行时监测模块持续检测通信过程中掉队通信流的产生情况，并将监测结果反馈至调度器，为调度策略的动态优化提供依据。 仿真实验以 Crux 和 Gsched 的优先级配置策略作为对照方法，以训练通信阶段时长和集群计算资源利用率作为评价指标。实验结果表明，相较于Crux，所提方法可使训练通信阶段时长降低18.4%，集群计算资源利用率提高10.4%；相较于 Gsched，训练通信阶段时长降低13.5%，集群计算资源利用率提高6.3%。此外，通过消融实验、算法近似比分析实验以及最佳掉队阈值分析实验，进一步验证了流优先级配置、路径选择和反馈机制对算法性能提升的有效性，并确定ρ=20%时能够取得较优的综合调度效果。

Abstract: As the number of training jobs in distributed deep neural network clusters continues to grow, communication overhead has become a significant factor affecting training completion time, making network sharing among multiple concurrent training jobs increasingly critical. Existing studies have mainly focused on distributed training communication optimization and network resource scheduling by employing task-level scheduling or single-job communication optimization to reduce communication overhead. However, task-level scheduling cannot differentiate the priorities of communication flows within the same training job, while single-job communication optimization fails to adequately address the competition among communication flows from multiple concurrent training jobs. Consequently, these approaches are unable to effectively mitigate the straggler effect caused by delayed critical communication flows during synchronous communication, leading to inefficient utilization of network resources. To address these challenges, a multi-job communication flow scheduling method based on flow-granularity priority assignment and path selection joint optimization is proposed. First, the network resource competition problem among multiple Ring-AllReduce training jobs is formulated as a mathematical optimization model. The model takes the priority assignment and path selection of all communication flows as input and estimates the throughput of each flow under the assumption of equal bandwidth sharing among flows within the same priority queue. The communication completion time of each training job is then estimated, and the overall communication-phase duration of all training jobs is minimized as the optimization objective. Since jointly optimizing the priority and routing configurations of all communication flows leads to exponential computational complexity, a heuristic solution combining local optimal scheduling and a runtime feedback mechanism is developed. By integrating binary search and backtracking techniques, the proposed algorithm significantly reduces computational complexity while maintaining high scheduling accuracy. During execution, the runtime feedback mechanism dynamically adjusts flow priorities according to detected straggler flows, thereby further alleviating the straggler effect in synchronous communication. To evaluate the effectiveness of the proposed method, a distributed training simulation platform based on container technology and virtual switches is developed to emulate both computing nodes and network switches. A communication flow scheduling system consisting of a centralized scheduler, node agents, and runtime monitoring modules is implemented. The centralized scheduler computes the priority assignment and routing decisions for communication flows and dynamically updates node traffic control policies and switch forwarding rules through remote procedure calls (RPC) and the OpenFlow protocol, respectively. Meanwhile, the runtime monitoring module continuously detects straggler communication flows during training and reports the monitoring results to the centralized scheduler, enabling adaptive optimization of scheduling decisions. Simulation experiments compare the proposed method with the priority assignment strategies of Crux and Gsched using communication-phase duration and cluster computational resource utilization as evaluation metrics. Experimental results demonstrate that, compared with Crux, the proposed method reduces communication-phase duration by 18.4% and improves cluster computational resource utilization by 10.4%. Compared with Gsched, it reduces communication-phase duration by 13.5% and improves computational resource utilization by 6.3%. In addition, ablation studies, approximation ratio analysis, and straggler-threshold sensitivity analysis further verify the effectiveness of the proposed flow priority assignment, path selection, and runtime feedback mechanisms. The results also indicate that a straggler threshold of ρ = 20% achieves the relatively good overall scheduling performance.