基于图神经网络的多智能体强化学习值函数分解方法

doi:10.19678/j.issn.1000-3428.0067919

计算机工程 ›› 2024, Vol. 50 ›› Issue (5): 62-70. doi: 10.19678/j.issn.1000-3428.0067919

基于图神经网络的多智能体强化学习值函数分解方法

孙文洁¹, 李宗民^1,2, 孙浩淼¹

1. 中国石油大学(华东)计算机科学与技术学院, 山东青岛 266580;
2. 山东石油化工学院大数据与基础科学学院, 山东东营 257061

收稿日期:2023-06-25 修回日期:2023-08-10 出版日期:2024-05-15 发布日期:2024-05-23
通讯作者: 孙文洁,E-mail:sunwenjie@s.upc.edu.cn E-mail:sunwenjie@s.upc.edu.cn
基金资助:
国家重点研发计划(2019YFF0301800);国家自然科学基金(61379106);山东省自然科学基金(ZR2013FM036, ZR2015FM011)。

Multi-Agent Reinforcement Learning Value Function Factorization Approach Based on Graph Neural Network

SUN Wenjie¹, LI Zongmin^1,2, SUN Haomiao¹

1. School of Computer Science and Technology, China University of Petroleum(East China), Qingdao 266580, Shandong, China;
2. College of Big Data and Basic Science, Shandong Institute of Petroleum and Chemical Technology, Dongying 257061, Shandong, China

Received:2023-06-25 Revised:2023-08-10 Online:2024-05-15 Published:2024-05-23
Contact: 孙文洁,E-mail:sunwenjie@s.upc.edu.cn E-mail:sunwenjie@s.upc.edu.cn

摘要/Abstract

摘要： 如何在部分可观测的情况下实现智能体之间的协同配合是多智能体强化学习(MARL)中的一个重要问题。值函数分解方法解决了信用分配问题,是一种实现多智能体之间协同配合的有效方法,然而在现有的值函数分解方法中,智能体个体动作值函数仅取决于局部信息,不允许智能体之间进行显式的信息交换,阻碍了这一系列算法的性能,使其无法适用于复杂场景。为了解决这一问题,在值函数分解方法中引入智能体间的通信,为智能体提供有效的非局部信息以帮助其理解复杂环境。在此基础上,提出一个基于图神经网络的分层通信模型,通过图神经网络提取相邻智能体之间需要交换的有用信息,同时模型能够实现从非通信向充分通信过渡,在通信范围有限的情况下实现全局合作,适用于现实世界中通信范围受约束的情况。在星际争霸Ⅱ多智能体挑战赛(SMAC)环境和捕食者-猎物(PP)环境下进行实验,结果表明,在SMAC的4个不同场景下,该方法与QMIX、VBC等基线算法相比平均胜率提升2~40个百分点,并且能够有效解决非单调环境下的捕食者-猎物问题。

关键词: 深度强化学习, 多智能体环境, 智能体协同, 智能体通信, 图神经网络

Abstract: Collaborative cooperation between agents in partially observable situations is an important problem in Multi-Agent Reinforcement Learning(MARL). The value function factorization approach solves the credit assignment problem and effectively achieves collaborative cooperation between agents. However, existing value function factorization approaches depend only on individual value functions with local information and do not allow explicit information exchange between agents, making them unsuitable for complex scenarios. To address this problem, this study introduces communication in the value function factorization approach to provide effective nonlocal information to agents, helping them understand complex environments. Furthermore, unlike existing communication approaches, the proposed approach uses a multi-layer message passing architecture based on Graph Neural Network(GNN), which extracts useful information that must be exchanged between neighboring agents. Simultaneously, the model realizes the transition from non-communication to full communication and achieves global cooperation with a limited communication range, which is suitable for real-world applications where the communication range is constrained. The results of experiments in the StarCraft II Multi-Agent Challenge(SMAC) and Predator-Prey(PP) environments demonstrate that the average winning rate of this approach improves by 2-40 percentage points compared with those of baseline algorithms, such as QMIX and VBC, in four different scenarios of SMAC. Furthermore, the proposed approach effectively solves the PP problem in non-monotonic environments.

Key words: deep reinforcement learning, multi-agent environment, agent cooperation, agent communication, Graph Neural Network(GNN)

中图分类号:

TP18

孙文洁, 李宗民, 孙浩淼. 基于图神经网络的多智能体强化学习值函数分解方法[J]. 计算机工程, 2024, 50(5): 62-70.

SUN Wenjie, LI Zongmin, SUN Haomiao. Multi-Agent Reinforcement Learning Value Function Factorization Approach Based on Graph Neural Network[J]. Computer Engineering, 2024, 50(5): 62-70.

https://www.ecice06.com/CN/Y2024/V50/I5/62

参考文献

[1] 黄子蓉. 基于深度强化学习的多智能体协同研究[D]. 太原:太原理工大学, 2021. HUANG Z R. Research of multi-agent cooperation based on deep reinforcement learning[D]. Taiyuan:Taiyuan University of Technology, 2021. (in Chinese)
[2] 侯胜男. 基于强化学习的多智能体协同对抗算法[D]. 杭州:浙江大学, 2022. HOU S N. Multi-agent confrontation algorithm based on reinforcement learning[D].Hangzhou:Zhejiang University, 2022. (in Chinese)
[3] 李雪松, 张锲石, 宋呈群, 等. 自动驾驶场景下的轨迹预测技术综述[J]. 计算机工程, 2023, 49(5):1-11. LI X S, ZHANG Q S, SONG C Q, et al. Review of trajectory prediction technology in autonomous driving scenes[J]. Computer Engineering, 2023, 49(5):1-11.(in Chinese)
[4] 熊丽琴, 曹雷, 赖俊, 等. 基于值分解的多智能体深度强化学习综述[J]. 计算机科学, 2022, 49(9):172-182. XIONG L Q, CAO L, LAI J, et al. Overview of multi-agent deep reinforcement learning based on value factorization[J]. Computer Science, 2022, 49(9):172-182.(in Chinese)
[5] ZHANG S Q, ZHANG Q, LIN J. Efficient communication in multi-agent reinforcement learning via variance based control[C]//Proceedings of the Annual Conference on Neural Information Processing Systems. Cambridge, USA:MIT Press, 2019:5471-5483.
[6] YUAN L, WANG J H, ZHANG F X, et al. Multi-agent incentive communication via decentralized teammate modeling[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2022, 36(9):9466-9474.
[7] WU B, YANG X Y, SUN C X, et al. Learning effective value function factorization via attentional communication[C]//Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics. Washington D. C., USA:IEEE Press, 2020:629-634.
[8] RASHID T, FARQUHAR G, PENG B, et al. Weighted QMIX:expanding monotonic value function factorisation for deep multi-agent reinforcement learning[C]//Proceedings of the Annual Conference on Neural Information Processing Systems. Cambridge, USA:MIT Press, 2020:10199-10210.
[9] OLIEHOEK F A, AMATO C. A concise introduction to decentralized POMDPs[M]. Berlin, Germany:Springer, 2016.
[10] SON K, KIM D, KANG W J, et al. QTRAN:learning to factorize with transformation for cooperative multi-agent reinforcement learning[C]//Proceedings of International Conference on Machine Learning. Washington D.C., USA:ACM Press, 2019:5887-5896.
[11] TAN M. Multi-agent reinforcement learning:independent vs. cooperative agents[C]//Proceedings of International Conference on Machine Learning.[S.l.]:Elsevier, 1993:330-337.
[12] OLIEHOEK F A, SPAAN M T J, VLASSIS N. Optimal and approximate Q-value functions for decentralized POMDPs[J]. Journal of Artificial Intelligence Research, 2008, 32:289-353.
[13] 邓晖奕, 李勇振, 尹奇跃. 引入通信与探索的多智能体强化学习QMIX算法[J]. 计算机应用, 2023, 43(1):202-208. DENG H Y, LI Y Z, YIN Q Y. Improved QMIX algorithm from communication and exploration for multi-agent reinforcement learning[J]. Journal of Computer Applications, 2023, 43(1):202-208.(in Chinese)
[14] SUNEHAG P, LEVER G, GRUSLYS A, et al. Value-decomposition networks for cooperative multi-agent learning based on team reward[C]//Proceedings of the 17th International Conference on Autonomous Agents and Multiagent Systems. New York, USA:ACM Press, 2018:2085-2087.
[15] RASHID T, SAMVELYAN M, WITT C D. QMIX:monotonic value function factorisation for deep multi-agent reinforcement learning[C]//Proceedings of International Conference on Learning Representations. Washington D. C., USA:IEEE Press, 2018:1-14.
[16] WANG J, REN Z, LIU T, YU Y, et al. QPLEX:duplex dueling multi-agent Q-learning[C]//Proceedings of International Conference on Learning Representations. Washington D. C., USA:IEEE Press, 2021:301-310.
[17] 宋健, 王子磊. 基于值分解的多目标多智能体深度强化学习方法[J]. 计算机工程, 2023, 49(1):31-40. SONG J, WANG Z L. Multi-goal multi-agent deep reinforcement learning method based on value decomposition[J]. Computer Engineering, 2023, 49(1):31-40. (in Chinese)
[18] SHEN S, QIU M, LIU J, et al. ResQ:a residual q function-based approach for multi-agent reinforcement learning value factorization[C]//Proceedings of the Annual Conference on Neural Information Processing Systems. Cambridge, USA:MIT Press, 2022:457-464.
[19] DAS A, GERVET T, ROMOFF J, et al. TarMAC:targeted multi-agent communication[C]//Proceedings of International Conference on Machine Learning. Washington D. C., USA:ACM Press, 2019:1538-1546.
[20] SINGH A, JAIN T, SUKHBAATAR S. Learning when to communicate at scale in multiagent cooperative and competitive tasks[EB/OL].[2023-05-02]. https://arxiv.org/abs/1812.09755.
[21] LIU Y, WANG W X, HU Y J, et al. Multi-agent game abstraction via graph attention neural network[J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(5):7211-7218.
[22] SUKHBAATAR S, SZLAM A, FERGUS R. Learning multiagent communication with backpropagation[C]//Proceedings of the Annual Conference on Neural Information Processing Systems. Cambridge, USA:MIT Press, 2016:2252-2260.
[23] JIANG J, DUN C, HUANG T, et al. Graph convolutional reinforcement learning[C]//Proceedings of International Conference on Learning Representations. Washington D. C., USA:IEEE Press, 2020:25-30.
[24] HAUSKNECHT M, STONE P. Deep recurrent Q-learning for partially observable MDPs[EB/OL].[2023-05-02]. https://arxiv.org/abs/1507.06527v1.
[25] WANG Y, SUN Y B, LIU Z W, et al. Dynamic graph CNN for learning on point clouds[J]. ACM Transactions on Graphics, 38(5):146.

选择文件类型/文献管理软件名称

选择包含的内容

基于图神经网络的多智能体强化学习值函数分解方法

Multi-Agent Reinforcement Learning Value Function Factorization Approach Based on Graph Neural Network

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	李泽霖, 吕兆峰, 陈富强, 李克. 基于多跳信息融合的实体对齐模型[J]. 计算机工程, 2024, 50(9): 142-152.
[2]	石琼, 段辉, 师智斌. 基于深度强化学习的可信任务卸载方案[J]. 计算机工程, 2024, 50(8): 142-152.
[3]	何杏宇, 周易歆, 罗东旭, 杨桂松. 基于图神经网络和多主体评价的教学资源推荐[J]. 计算机工程, 2024, 50(7): 13-22.
[4]	张正康, 杨丹, 聂铁铮, 寇月. 基于图结构聚类的自监督学习疾病诊断方法[J]. 计算机工程, 2024, 50(7): 360-371.
[5]	游奔, 李晓红, 姚锦, 冯绍杰. 基于多粒度图与注意力机制的半监督短文本分类[J]. 计算机工程, 2024, 50(5): 83-90.
[6]	傅明建, 郭福强. 基于深度强化学习的无信号灯路口决策研究[J]. 计算机工程, 2024, 50(5): 91-99.
[7]	张宝鑫, 杨丹, 聂铁铮, 寇月. 基于自监督的多视角图协同过滤推荐方法[J]. 计算机工程, 2024, 50(5): 100-110.
[8]	吴正洋, 张广涛, 黄立, 汤庸. 基于异质图嵌入和会话交互的课程推荐模型[J]. 计算机工程, 2024, 50(4): 95-103.
[9]	王华珍, 许泽, 孙悦, 丘斌, 陈坚, 邱强斌. 基于事件演化图的多标记事件预测模型[J]. 计算机工程, 2024, 50(4): 132-140.
[10]	王俊, 赖会霞, 万玥, 张仕. 基于角度的图神经网络高维数据异常检测方法[J]. 计算机工程, 2024, 50(3): 156-165.
[11]	王柏涵, 姜晓燕, 范柳伊. 基于深度监督隐空间构建的语义分割改进方法[J]. 计算机工程, 2024, 50(3): 191-199.
[12]	杜海军, 余粟. 基于时空图注意力网络的服务机器人动态避障[J]. 计算机工程, 2024, 50(2): 105-112.
[13]	李伟健, 胡慧君. 基于潜在特征增强网络的视频描述生成方法[J]. 计算机工程, 2024, 50(2): 266-272.
[14]	顾宇衡, 潘嘉诚, 钱江波, 董一鸿. 阿尔茨海默病的图神经网络分类方法研究进展[J]. 计算机工程, 2024, 50(10): 35-50.
[15]	蔡梓越, 谭北海, 余荣, 黄旭民, 王思明. 面向6G物联网设备协同的区块链动态分片[J]. 计算机工程, 2024, 50(1): 50-59.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于图神经网络的多智能体强化学习值函数分解方法

Multi-Agent Reinforcement Learning Value Function Factorization Approach Based on Graph Neural Network

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

本文评价