基于值分解的多目标多智能体深度强化学习方法

doi:10.19678/j.issn.1000-3428.0063303

摘要/Abstract

摘要： 多智能体深度强化学习方法可应用于真实世界中需要多方协作的场景，是强化学习领域内的研究热点。在多目标多智能体合作场景中，各智能体之间具有复杂的合作与竞争并存的混合关系，在这些场景中应用多智能体强化学习方法时，其性能取决于该方法是否能够充分地衡量各智能体之间的关系、区分合作和竞争动作，同时也需要解决高维数据的处理以及算法效率等应用难点。针对多目标多智能体合作场景，在QMIX模型的基础上提出一种基于目标的值分解深度强化学习方法，并使用注意力机制衡量智能体之间的群体影响力，利用智能体的目标信息实现量两阶段的值分解，提升对复杂智能体关系的刻画能力，从而提高强化学习方法在多目标多智能体合作场景中的性能。实验结果表明，相比QMIX模型，该方法在星际争霸2微观操控平台上的得分与其持平，在棋盘游戏中得分平均高出4.9分，在多粒子运动环境merge和cross中得分分别平均高出25分和280.4分，且相较于主流深度强化学习方法也具有更高的得分与更好的性能表现。

关键词: 深度强化学习, 多智能体, 多目标, 值分解, 注意力机制

Abstract: Multi-agent deep reinforcement learning method can be used in scenarios that require multi-party cooperation in the real world, which remains a challenge in the field of reinforcement learning.In these scenarios, agents usually have complex relationships with each other, including both cooperation and competition.The performance of multi-agent reinforcement learning method depends on whether the method can correctly assess the relationships among agents and distinguish cooperative and competitive actions.Additionally, it also faces the efficiency problem related to high-dimensional data processing.Focused on multi-goal multi-agent cooperation scenarios, this paper proposes a deep multi-goal multi-agent reinforcement learning method, using value function factorization based on QMIX. In this method, agents' goals and the attention mechanism are utilized to measure the social influence among them, which leads to an improved characterization of complexity of their relationships.The experimental results show that the reward scores are almost the same, 4.9 higher, 25 higher and 280.4 higher than QMIX's in StarCraft 2 platform, checker game, and the merge and cross map of multi-particle environment.The method presented here shows higher scores and better performance compared to other mainstream deep reinforcement learning methods in the representative scenarios.

Key words: deep reinforcement learning, multi-agent, multi-goal, value decomposition, attention mechanism

中图分类号:

TP391

宋健, 王子磊. 基于值分解的多目标多智能体深度强化学习方法[J]. 计算机工程, 2023, 49(1): 31-40.

SONG Jian, WANG Zilei. Multi-Goal Multi-Agent Deep Reinforcement Learning Method Based on Value Decomposition[J]. Computer Engineering, 2023, 49(1): 31-40.

https://www.ecice06.com/CN/Y2023/V49/I1/31

图/表 10

20230701174841

20230701174844

20230701174847

20230701174850

20230701174854

20230701174857

20230701174901

20230701174905

20230701174908

20230701174912

参考文献

[1] YANG J C, NAKHAEI A, ISELE D, et al.CM3:cooperative multi-goal multi-stage multi-agent reinforcement learning[EB/OL].[2021-10-20].https://arxiv.org/abs/1809.05188.
[2] TAN M.Multi-agent reinforcement learning:independent vs.cooperative agents[C]//Proceedings of the 20th ACM International Conference on Machine Learning.Washington D.C., USA:ACM Press, 1993:330-337.
[3] LOWE R, WU Y, TAMAR A, et al.Multi-agent actor-critic for mixed cooperative-competitive environments[C]//Proceedings of Advances in Neural Information Processing Systems.Cambridge, USA:MIT Press, 2017:6382-6393.
[4] LI S H, WU Y, CUI X Y, et al.Robust multi-agent reinforcement learning via minimax deep deterministic policy gradient[J].Artificial Intelligence, 2019, 33(1):4213-4220.
[5] RASHID T, SAMVELYAN M, WITT C D.QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning[C]//Proceedings of International Conference on Learning Representations.Washington D.C., USA:IEEE Press, 2018:1-14.
[6] LITTMAN M L.Markov games as a framework for multi-agent reinforcement learning[M].Amsterdam, Netherlands:Elsevier, 1994.
[7] TAMPUU A, MATIISEN T, KODELJA D, et al.Multi-agent cooperation and competition with deep reinforcement learning[J].PLoS One, 2017, 12(4):0172395.
[8] MORDATCH I, ABBEEL P.Emergence of grounded compositional language in multi-agent populations[J].Artificial Intelligence, 2018, 32(1):1-16.
[9] PANAIT L, LUKE S A.Cooperative multi-agent learning:the state of the art[J].Autonomous Agents and Multi-Agent Systems, 2005, 11(3):387-434.
[10] MNIH V, KAVUKCUOGLU K, SILVER D, et al.Human-level control through deep reinforcement learning[J].Nature, 2015, 518(7540):529-533.
[11] OMIDSHAFIEI S, PAZIS J, AMATO C, et al.Deep decentralized multi-task multi-agent reinforcement learning under partial observability[EB/OL].[2021-10-20].https://arxiv.org/abs/1703.06182.
[12] FOERSTER J, NARDELLI N, FARQUHAR G, et al.Stabilising experience replay for deep multi-agent reinforcement learning[C]//Proceedings of the 34th International Conference on Machine Learning.Washington D.C., USA:IEEE Press, 2017:1146-1155.
[13] GUESTRIN C, KOLLER D, PARR R.Multi-agent planning with factored MDPs[C]//Proceedings of Advances in Neural Information Processing Systems.Cambridge, USA:MIT Press, 2002:1523-1530.
[14] FANG M, GROEN F C A.Collaborative multi-agent reinforcement learning based on experience propagation[J].Journal of Systems Engineering and Electronics, 2013, 24(4):683-689.
[15] SINGH A, JAIN T, SUKHBAATAR S.Learning when to communicate at scale in multiagent cooperative and competitive tasks[EB/OL].[2021-10-20].https://arxiv.org/abs/1812.09755.
[16] AUSTERWEIL J L, BRAWNER S, GREENWALD A, et al.How other-regarding preferences can promote cooperation in nonzero-sum grid games[C]//Proceedings of AAAI Symposium on Challenges and Opportunities in Multi-agent Learning for Real World.[S.1.]:AAAI Press, 2016:1-18.
[17] FOERSTER J, FARQUHAR G, AFOURAS T, et al.Counterfactual multi-agent policy gradients[J].Artificial Intelligence, 2018, 32(1):1-14.
[18] GUPTA J K, EGOROV M, KOCHENDERFER M.Cooperative multi-agent control using deep reinforcement learning[C]//Proceedings of International Conference on Autonomous Agents and Multiagent Systems.Washington D.C., USA:IEEE Press, 2017:66-83.
[19] SUNEHAG P, LEVER G, GRUSLYS A, et al.Value-decomposition networks for cooperative multi-agent learning based on team reward[C]//Proceedings of the 17th International Conference on Autonomous Agents and Multi-agent Systems.Berlin, Germany:Springer, 2017:1-18.
[20] ZHANG K Q, YANG Z R, LIU H, et al.Fully decentralized multi-agent reinforcement learning with networked agents[EB/OL].[2021-10-20].https://arxiv.org/abs/1802.08757.
[21] OLIEHOEK F A, SPAAN M T J, VLASSIS N.Optimal and approximate Q-value functions for decentralized POMDPs[J].Journal of Artificial Intelligence Research, 2008, 32(3):289-353.
[22] OLIEHOEK F A, AMATO C.A concise introduction to decentralized POMDPs[M].Berlin, Germany:Springer, 2016.
[23] HAUSKNECHT M, STONE P.Deep recurrent Q-learning for partially observable MDPs[C]//Proceedings of AAAI Fall Symposium on Sequential Decision Making for Intelligent Agents.[S.1.]:AAAI Press, 2015:1-13.
[24] HOCHREITER S, SCHMIDHUBER J.Long short-term memory[J].Neural Computation, 1997, 9(8):1735-1780.
[25] CHUNG J, GULCEHRE C, CHO K, et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[C]//Proceedings of Advances in Neural Information Processing Systems.Cambridge, USA:MIT Press, 2014:1-15.
[26] CHUNG J, GULCEHRE C, CHO K, et al.Empirical evaluation of gated recurrent neural networks on sequence modeling[EB/OL].[2021-10-20].https://arxiv.org/abs/1412.3555.
[27] DUGAS C, BENGIO Y, BLISLE F, et al.Incorporating functional knowledge in neural networks[J].Journal of Machine Learning Research, 2009, 10(3):1239-1262.
[28] HA D, DAI A, LE Q V.Hyper networks[C]//Proceedings of International Conference on Learning Representations.Washington D.C., USA:IEEE Press, 2017:1-17.
[29] VINYALS O, EWALDS T, BARTUNOV S, et al.StarCraft II:a new challenge for reinforcement learning[EB/OL].[2021-10-20].https://arxiv.org/abs/1708.04782.

选择文件类型/文献管理软件名称

选择包含的内容