Abstract: The Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm alleviates the problem of environmental nonstationarity by introducing global information when solving multi-agent path planning problems. However, in complex environments, Multi-Agent Reinforcement Learning (MARL) algorithms suffer from challenges such as sparse rewards and low levels of agent collaboration. To address these issues, a multi-agent path planning based on state-action prediction, called SA-MADDPG, is proposed. First, a novelty reward module based on a Long Short-Term Memory (LSTM) network is designed to provide novelty rewards to agents without relying on current observations and actions, thereby alleviating the reward sparsity problem. Second, an action prediction module is designed by explicitly incorporating collaborative information. Additionally, a dynamic weight term based on Q-value gain is introduced to guide agents in balancing the optimization of their individual task strategies with the optimization of collaborative task strategies, thus enhancing the level of inter-agent collaboration. Finally, a three-dimensional multi-agent path planning simulation environment based on drones is constructed to comprehensively evaluate the performance of the proposed algorithm. Experimental results show that, in the obstacle density experiment, the average reward of SA-MADDPG increases by 5.26%—15.81% and the average episode time decreases by 10.96%—16.05%; in the agent number experiment, the average reward increases by 16.32%— 22.90% and the average episode time decreases by 15.03%—25.15%.

Key words: path planning, deep reinforcement learning, multi-agent, three-dimensional simulation environment, agent collaboration

摘要： 多智能体深度确定性策略梯度(MADDPG)算法在解决多智能体路径规划问题时，通过引入全局信息缓解了环境非平稳性问题。然而，在复杂环境下，多智能体强化学习（MARL）算法仍存在奖励稀疏、智能体协作水平低下等缺陷。为解决上述问题，提出了一种基于状态动作预测的多智能体路径规划算法SA-MADDPG。首先，设计了基于长短期记忆(LSTM)网络的新奇奖励模块，能够在不依赖当前观测和动作的情况下给予智能体新奇奖励值，以缓解奖励稀疏问题；然后，设计了一个动作预测模块，通过显式地引入协作信息，并提出了一个基于Q值增益的动态权重项，指导智能体权衡自身任务策略优化与协作任务策略优化，以提升智能体协作水平；最后，构建了一个基于无人机的三维多智能体路径规划仿真环境，以综合评估提出算法的性能。实验结果表明：在障碍物密度实验中，SA-MADDPG算法的平均奖励提高5.26%~15.81%，平均回合时间减少10.96%~16.05%；在智能体数量实验中，平均奖励提高16.32%~22.90%，平均回合时间减少15.03%~25.15%。

关键词: 路径规划, 深度强化学习, 多智能体, 三维仿真环境, 智能体协作

CLC Number: 

• TP391