基于交叉注意力扩散模型的对手建模

doi:10.19678/j.issn.1000-3428.0070737

摘要/Abstract

摘要：

对手建模作为多智能体博弈对抗的关键技术, 其目的为学习对手的行为以减少环境的不确定性并帮助决策。然而, 现有的对手建模方法大多采用离线训练加在线适应的结构, 在离线训练中采用传统神经动力学模型一步步预测受控智能体的行为, 容易形成单步误差进而形成累积误差, 且在在线适应中面对未知对手时, 亦会导致受控智能体计划状态偏离数据集分布。为解决上述问题, 提出基于扩散模型并利用交叉注意力和对手建立关联的方法。该方法利用扩散模型可以同时生成多步规划序列这一特点解决了累积偏差问题, 同时提出策略集的概念, 通过在线微调的方式不仅解决了计划偏离问题, 而且也解决了在线训练初始阶段会破坏离线策略的问题。在开放的密集奖励和稀疏奖励的竞争环境中的实验结果均充分证明了该方法卓越的性能。

关键词: 对手建模, 多智能体博弈对抗, 扩散模型, 交叉注意力机制, 在线微调

Abstract:

As a key technology in multi-agent game confrontation, opponent modeling aims to learn the behavior of the opponent to reduce the uncertainty of the environment and facilitate decision-making. However, most existing opponent modeling methods adopt the structure of offline training and online adaptation. During offline training, traditional neural dynamics model is used to predict the agent step-by-step, which facilitates the formation of single-step and cumulative errors. Additionally, when facing an unknown opponent during online adaptation, the planned states of the controlled agent deviate from the distribution of the dataset. To solve these problems, a method based on a diffusion model and cross-attention is proposed to establish a correlation with the opponent. The cumulative bias problem is solved using the feature that the diffusion model can generate multistep planning sequences simultaneously. The concept of a strategy set is proposed, and the deviation problem is not only solved by online fine-tuning but also prevents the problem that the offline strategy will be destroyed in the initial stages of online training. Experimental results in both open intensive reward and sparse reward competitive environments demonstrate the superior performance of this method.

Key words: opponent modeling, multi-agent game confrontation, diffusion model, cross-attention mechanism, online fine-tuning

何明研, 李斯源, 刘鹏, 黄剑华. 基于交叉注意力扩散模型的对手建模[J]. 计算机工程, 2026, 52(7): 76-85.

HE Mingyan, LI Siyuan, LIU Peng, HUANG Jianhua. Opponent Modeling Based on Cross-Attention Diffusion Model[J]. Computer Engineering, 2026, 52(7): 76-85.

https://www.ecice06.com/CN/Y2026/V52/I7/76

图/表 17

图1 总体框架图

Fig.1 General frame diagram

图2 对手策略编码器

Fig.2 Opponent policy encoder

图3 融合解码器

Fig.3 Fusion decoder

图4 在线微调阶段

Fig.4 Online fine-tuning stage

图5 MS环境

Fig.5 MS environment

图6 PA环境

Fig.6 PA environment

图7 PA环境下与TAO对比

Fig.7 Comparison with TAO in PA environment

参考文献 37

1	FOERSTER J N, CHEN R Y, AL-SHEDIVAT M, et al. Learning with opponent-learning awareness[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1709.04326.
2	HE H, BOYD-GRABER J, KWOK K, et al. Opponent modeling in deep reinforcement learning[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1609.05559.
3	KIM D K, LIU M, RIEMER M D, et al. A policy gradient algorithm for learning to learn in multiagent reinforcement learning[C]//Proceedings of International Conference on Machine Learning. [S. l.]: PMLR, 2021: 5541-5550.
4	王腾, 黄俊松, 王乐庭, 等. 基于MADDPG的多阵面相控阵雷达引导搜索资源优化算法. 计算机工程, 2024, 50(11): 38- 48. doi: 10.19678/j.issn.1000-3428.0069838
	WANG T, HUANG J S, WANG L T, et al. Multi-antenna phased array radar-guided search resource optimization algorithm based on MADDPG. Computer Engineering, 2024, 50(11): 38- 48. doi: 10.19678/j.issn.1000-3428.0069838
5	施伟, 冯旸赫, 程光权, 等. 基于深度强化学习的多机协同空战方法研究. 自动化学报, 2021, 47(7): 1610- 1623.
	SHI W, FENG Y H, CHENG G Q, et al. Research on multi-aircraft cooperative air combat method based on deep reinforcement learning. Acta Automatica Sinica, 2021, 47(7): 1610- 1623.
6	JING Y, LI K, LIU B, et al. Towards offline opponent modeling with in-context learning[C]//Proceedings of the 12th International Conference on Learning Representations. New York, USA: ACM Press, 2023: 1-13.
7	白天, 吕璐瑶, 李储, 等. 基于深度强化学习的游戏智能引导算法. 吉林大学学报(理学版), 2025, 63(1): 91- 98. doi: 10.13413/j.cnki.jdxblxb.2023555
	BAI T, LÜ L Y, LI C, et al. Game intelligent guidance algorithm based on deep reinforcement learning. Journal of Jilin University (Science Edition), 2025, 63(1): 91- 98. doi: 10.13413/j.cnki.jdxblxb.2023555
8	JIANG H B, JIANG J C, LU Z Q, et al. Model-based opponent modeling[C]//Proceedings of the Advances in Neural Information Processing Systems. New Orleans, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022: 28208-28221.
9	徐浩添, 秦龙, 曾俊杰, 等. 基于深度强化学习的对手建模方法研究综述. 系统仿真学报, 2023, 35(4): 671- 694. doi: 10.16182/j.issn1004731x.joss.22-0555
	XU H T, QIN L, ZENG J J, et al. Research progress of opponent modeling based on deep reinforcement learning. Journal of System Simulation, 2023, 35(4): 671- 694. doi: 10.16182/j.issn1004731x.joss.22-0555
10	LI S, ZHAO H D. A survey on representation learning for user modeling[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence. Yokohama, Japan: International Joint Conferences on Artificial Intelligence Organization, 2020: 4997-5003.
11	ROSMAN B, HAWASLY M, RAMAMOORTHY S. Bayesian policy reuse. Machine Learning, 2016, 104(1): 99- 127. doi: 10.1007/s10994-016-5547-y
12	HERNANDEZ-LEAL P, TAYLOR M E, ROSMAN B, et al. Identifying and tracking switching, non-stationary opponents: a Bayesian approach[C]//Proceedings of the AAAI Workshop on Multiagent Interaction without Prior Coordination. Palo Alto, USA: AAAI Press, 2016: 1-15.
13	ZHENG Y, MENG Z P, HAO J Y, et al. A deep Bayesian policy reuse approach against non-stationary agents[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2018: 962-997.
14	YANG T P, MENG Z P, HAO J Y, et al. Towards efficient detection and optimal response against sophisticated opponents[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1809.04240.
15	HONG Z W, SU S Y, SHANN T Y, et al. A deep policy inference Q-network for multi-agent systems[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1712.07893.
16	AL-SHEDIVAT M, BANSAL T, BURDA Y, et al. Continuous adaptation via meta-learning in nonstationary and competitive environments[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1710.03641.
17	SOHL-DICKSTEIN J, WEISS E A, MAHESWARANATHAN N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1503.03585.
18	ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 10674-10685.
19	KONG Z F, PING W, HUANG J J, et al. DiffWave: a versatile diffusion model for audio synthesis[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2009.09761.
20	陈子民, 关志涛. 基于条件扩散模型的图像分类对抗样本防御方法. 计算机工程, 2024, 50(12): 296- 305. doi: 10.19678/j.issn.1000-3428.0068512
	CHEN Z M, GUAN Z T. Image classification adversarial example defense method based on conditional diffusion model. Computer Engineering, 2024, 50(12): 296- 305. doi: 10.19678/j.issn.1000-3428.0068512
21	JANNER M, DU Y L, TENENBAUM J B, et al. Planning with diffusion for flexible behavior synthesis[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2205.09991.
22	WANG Z D, HUNT J J, ZHOU M Y. Diffusion policies as an expressive policy class for offline reinforcement learning[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2208.06193.
23	LU C, BALL P, TEH Y W, et al. Synthetic experience replay[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2303.06614.
24	XIAN Z, GKANATSIOS N, GERVET T, et al. ChainedDiffuser: unifying trajectory diffusion and keypose prediction for robotic manipulation[EB/OL]. [2024-11-14]. https://www.semanticscholar.org/paper/ChainedDiffuser%3A-Unifying-Trajectory-Diffusion-and-Xian-Gkanatsios/c36f3635e090aba84e5e83b904a7697e83730be6.
25	BAI C J, HE H R, LI X L, et al. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning[C]//Proceedings of the Advances in Neural Information Processing Systems. New Orleans, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 64896-64917.
26	ZHANG M Y, CAI Z A, PAN L, et al. MotionDiffuse: text-driven human motion generation with diffusion model[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2208.15001.
27	AJAY A, DU Y L, GUPTA A, et al. Is conditional generative modeling all you need for decision-making?[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2211.15657.
28	YANG Y, WANG J. An overview of multi-agent reinforcement learning from game theoretical perspective[EB/OL]. [2024-11-14]. https://arxiv.org/pdf/2011.00583v3.
29	ZHENG Q, ZHANG A, GROVER A. Online decision Transformer[C]//Proceedings of the International Conference on Machine Learning. [S. l.]: PMLR, 2022: 27042-27059.
30	HUSSEIN A, GABER M M, ELYAN E, et al. Imitation learning: a survey of learning methods. ACM Computing Surveys, 2018, 50(2): 1- 35. doi: 10.1145/3054912
31	AGRAWAL P, NAIR A, ABBEEL P, et al. Learning to poke by poking: experiential learning of intuitive physics[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1606.07419.
32	LANCTOT M, LOCKHART E, LESPIAU J B, et al. OpenSpiel: a framework for reinforcement learning in games[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1908.09453.
33	LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1706.02275.
34	PAPOUDAKIS G, CHRISTIANOS F, ALBRECHT S V. Agent modelling under partial observability for deep reinforcement learning[C]//Proceedings of the International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2021: 19210-19222.
35	ZINTGRAF L, DEVLIN S, CIOSEK K, et al. Deep interactive Bayesian reinforcement learning via meta-learning[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2101.03864.
36	SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[EB/OL]. [2024-11-14]. https://arxiv.org/abs/1707.06347.
37	PRAJAPAT M, AZIZZADENESHELI K, LINIGER A, et al. Competitive policy optimization[EB/OL]. [2024-11-14]. https://arxiv.org/abs/2006.10611.

[1]	赵昂, 相洁, 牛焱, 武旭斌, 宋子泽, 温昕. 基于磁共振成像的单图超分辨率扩散模型[J]. 计算机工程, 2026, 52(5): 259-269.
[2]	李健浪, 吴新电, 陈灵, 阳波, 唐文胜. 基于4D毫米波雷达与视觉融合的三维目标检测算法[J]. 计算机工程, 2026, 52(2): 299-310.
[3]	黎东丰, 陈雨人, 余博. 基于多层次特征融合的路面裂缝检测方法[J]. 计算机工程, 2026, 52(1): 154-165.
[4]	王兴杰, 王侃, 费蓉, 王怀军, 郭银波, 兰大鹏, 朱晓杰. 卫星边缘网络中基于扩散模型的算力分配策略[J]. 计算机工程, 2026, 52(1): 346-355.
[5]	姬莉霞, 周洪鑫, 肖士杰, 陈允峰, 张晗. 一种基于邻域注意力的扩散模型训练方法研究[J]. 计算机工程, 2025, 51(8): 262-269.
[6]	邸钦渤, 陈劭力, 时良仁. GRD: 基于GNN和扩散模型的多变量时序数据异常检测算法[J]. 计算机工程, 2025, 51(11): 35-44.
[7]	王皓嘉, 邓勇舰, 刘婷婷, 杨震. 基于DINO先验的混合特征面部表情识别模型[J]. 计算机工程, 2025, 51(10): 284-294.
[8]	陈子民, 关志涛. 基于条件扩散模型的图像分类对抗样本防御方法[J]. 计算机工程, 2024, 50(12): 296-305.
[9]	谭励,张哲,杨明华,胡计鹏. 基于加权虚拟力的空中传感器网络分段部署算法[J]. 计算机工程, 2016, 42(10): 118-123.
[10]	查那日苏, 何立强, 魏凤歧. 基于热扩散模型的测试程序分类[J]. 计算机工程, 2010, 36(11): 256-258,261.
[11]	张宁;杨树堂;陈健宁;陆松年. 一种以毒攻毒“益虫”的扩散模型研究[J]. 计算机工程, 2007, 33(06): 110-112.

选择文件类型/文献管理软件名称

选择包含的内容