基于角色学习的多智能体强化学习方法研究

doi:10.19678/j.issn.1000-3428.0070739

摘要/Abstract

摘要： 多智能体强化学习(MARL)在解决复杂协作任务中具有重要作用。然而，传统方法在动态环境和信息非平稳性方面存在显著局限性。针对这些挑战，本文提出了一种基于角色学习的多智能体强化学习框架(RoMAC)。该框架通过基于动作属性的角色划分，并借助角色分配网络实现智能体角色的动态分配，以提升多智能体协作效率。同时，框架采用分层通信设计，包括基于注意力机制的角色间通信和基于互信息的智能体间通信。在角色间通信中，RoMAC利用注意力机制生成高效的通信信息，以实现角色代理间的协调；在智能体间通信中，通过互信息生成有针对性的信息，从而提升角色组内部的决策质量。实验在星际争霸多智能体挑战(SMAC)环境中进行，结果表明，RoMAC胜率平均提高了约8.62%，收敛时间缩短了0.92百万步，通信负载平均降低了28.18%。消融实验进一步验证了RoMAC各模块在提升性能中的关键作用，体现了模型的稳健性与灵活性。综合实验结果表明，RoMAC在多智能体强化学习和协作任务中具有显著优势，为复杂任务的高效解决提供了可靠支持。

Abstract: Multi-agent reinforcement learning (MARL) plays a crucial role in solving complex cooperative tasks. However, traditional methods face significant limitations in dynamic environments and information non-stationarity. To address these challenges, this paper proposes a role-based multi-agent reinforcement learning framework (RoMAC). The framework employs role division based on action attributes and uses a role assignment network to dynamically allocate roles to agents, thereby enhancing the efficiency of multi-agent collaboration. Additionally, the framework adopts a hierarchical communication design, including inter-role communication based on attention mechanisms and inter-agent communication guided by mutual information. In inter-role communication, RoMAC leverages attention mechanisms to generate efficient communication messages for coordination between role delegates. In inter-agent communication, mutual information is used to produce targeted information, improving decision-making quality within role groups. Experiments conducted in the StarCraft Multi-Agent Challenge (SMAC) environment show that RoMAC achieves an average win rate improvement of approximately 8.62%, a reduction in convergence time by 0.92 million steps, and a 28.18% average decrease in communication load. Ablation studies further validate the critical contributions of each module in enhancing performance, showcasing the robustness and flexibility of the model. Overall, experimental results indicate that RoMAC offers significant advantages in multi-agent reinforcement learning and cooperative tasks, providing reliable support for efficiently addressing complex challenges.

沈思彤, 王耀吾, 谢在鹏, 唐斌. 基于角色学习的多智能体强化学习方法研究[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0070739.

SHEN Sitong, WANG Yaowu, XIE Zaipeng, TANG Bin. Research on Role-Based Multi-Agent Reinforcement Learning Methods[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0070739.

参考文献

[1] Zhu C, Dastani M, Wang S. A survey of multi-agent deep reinforcement learning with communication[J]. Autono-mous Agents and Multi-Agent Systems, 2024, 38(1): 4.
[2] Lowe R, Wu Y I, Tamar A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[J]. Advances in neural information processing systems, 2017, 30
[3] Rashid T, Samvelyan M, De Witt C S, et al. Monotonic value function factorisation for deep multi-agent reinforcement learning[J]. Journal of Machine Learning Research, 2020, 21(178): 1-51.
[4] 王涵, 俞扬, 姜远. 基于通信的多智能体强化学习进展综述[J]. 中国科学: 信息科学, 2022, 52(5): 742-764. WANG H, YU Y, JIANG Y. Review of the progress of communication-based multi-agent re-inforcement learning[J]. Science in China(Information Sciences) , 2022, 52(5): 742-764. (in Chinese)
[5] 罗彪,胡天萌,周育豪,等. 多智能体强化学习控制与决策研究综述 [J/OL]. 自动化学报 , 1-30[2024-12-14]. https://doi.org/10.16383/j.aas.c240392. LUO B, HU T M, ZHOU Y H, et al. Survey on Multi-agent Reinforcement Learning for Control and Decision-making [J/OL]. ACTA AUTOMATICA SINICA, 1-30[2024-12-14]. https://doi.org/10.16383/j.aas.c240392. (in Chinese)
[6] 丁世飞, 杜威, 张健, 等. 多智能体深度强化学习研究进展[J]. 计算机学报, 2024, 47(07): 1547-1567. DING S F, DU W, ZHANG J, et al. Research Progress of Multi-Agent Deep Reinforcement Learning[J]. Chinese Journal of Computers, 2024, 47(07): 1547-1567. (in Chinese)
[7] Mao H, Zhang Z, Xiao Z, et al. Learning agent communica-tion under limited bandwidth by message pruning[C] //Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(04): 5142-5149.
[8] Ding Z, Huang T, Lu Z. Learning individually inferred communication for multi-agent cooperation[J]. Advances in neural information processing systems, 2020, 33: 22069-22079.
[9] Wang Y, Zhong F, Xu J, et al. ToM2C: Target-oriented Multi-agent Communication and Cooperation with Theory of Mind[C]//The Tenth International Conference on Learning Representations, 2022.
[10] Hu S C, Shen L, Zhang Y, et al. Learning multi-agent communication from graph modeling perspective[C]// Proceedings of the 12th International Conference on Learning Representations. Vienna, Austria, 2024.
[11] Wang X, Li X, Shao J, et al. AC2C: Adaptively Controlled Two-Hop Communication for Multi-Agent Reinforcement Learning[C]//Proceedings of the 2023 International Conference on Autonomous Agents and Multiagent Systems, 2023: 427–435.
[12] Zhang S Q, Zhang Q, Lin J. Succinct and robust multi-agent communication with temporal message control[J]. Advances in neural information processing systems, 2020, 33: 17271-17282.
[13] Guan C, Chen F, Yuan L, et al. Efficient multi-agent communication via self-supervised information aggregati-on[J]. Advances in Neural Information Processing Systems, 2022, 35: 1020-1033.
[14] Kim D, Moon S, Hostallero D, et al. Learning to schedule communication in multi-agent reinforcement learning[C]// Proceedings of the 7th International Conference on Learn-ing Representations, 2019.
[15] Yuan L, Wang J, Zhang F, et al. Multi-agent incentive communication via decentralized teammate modeling[C]//Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(9): 9466-9474.
[16] Guo X, Shi D, Fan W. Scalable Communication for Multi-Agent Reinforcement Learning via Transformer-Based Email Mechanism[C]//Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, 2023: 126–134.
[17] Liu Z, Wan L, Sui X, et al. Deep Hierarchical Communication Graph in Multi-Agent Reinforcement Learning[C]//IJCAI, 2023: 208-216.
[18] Pina R, De Silva V, Artaud C, et al. Efficient Role-based Communication for Multi-Agent Systems[C]//Proceedings of the Autonomous Agents and Multi-Agent Systems, 2024.
[19] Duan W, Lu J, Xuan J, et al. Group-Aware Coordination Graph for Multi-Agent Reinforcement Learning[C] // Proceedings of the Thirty-Third International Joint Conference on Artificial Intelligence, 2024: 3926-3934.
[20] Wang T, Dong H, Lesser VR, et al. ROMA: Multi-Agent Reinforcement Learning with Emergent Roles[C]// Proceedings of the 37th International Conference on Machine Learning. 2020, 119: 9876-9886.
[21] Wang T, Gupta T, Mahajan A, et al. RODE: Learning Roles to Decompose Multi-Agent Tasks[C]//Proceedings of the 9th International Conference on Learning Representations. Proceedings of Machine Learning Research, 2021, 119: 9876-9886.
[22] Yang M, Zhao J, Hu X, et al. LDSA: Learning dynamic subtask assignment in cooperative multi-agent reinforce-ment learning[J]. Advances in Neural Information Process-ing Systems, 2022, 35: 1698-1710.
[23] Yang M, Zhao K, Wang Y, et al. Team-wise effective communication in multi-agent reinforcement learning[J]. Autonomous Agents and Multi-Agent Systems, 2024, 38(2): 36.
[24] Alemi AA, Fischer I, Dillon JV, et al. Deep Variational Information Bottleneck[C]//Proceedings of the 5th International Conference on Learning Representations, Toulon, France, Conference Track Proceedings, 2017.
[25] Samvelyan M, Rashid T, Schröder de Witt C, et al. The StarCraft Multi-Agent Challenge[C]//Proceedings of the 18th International Conference on Autonomous Agents and Multi-Agent Systems. 2019: 2186-2188.
[26] Zhang S Q, Zhang Q, Lin J. Efficient communication in multi-agent reinforcement learning via variance based control[J]. Advances in neural information processing systems, 2019, 32.

选择文件类型/文献管理软件名称

选择包含的内容