基于交叉注意力扩散模型的对手建模

doi:10.19678/j.issn.1000-3428.0070737

摘要/Abstract

摘要： 对手建模作为多智能体博弈对抗的关键技术,其目的为学习对手的行为以减少环境的不确定性并帮助决策。然而,现有的对手建模方法大多采用离线训练加在线适应的结构,在离线训练中采用传统神经动力学模型一步步预测受控智能体的行为,容易形成单步误差进而形成累积误差,且在在线适应中面对未知对手时,亦会导致受控智能体计划状态偏离数据集分布。为解决上述问题,提出基于扩散模型并利用交叉注意力和对手建立关联的方法。该方法利用扩散模型可以同时生成多步规划序列这一特点解决了累积偏差问题,同时提出策略集的概念,通过在线微调的方式不仅解决了计划偏离问题,而且也解决了在线训练初始阶段会破坏离线策略的问题。在开放的密集奖励和稀疏奖励的竞争环境中的实验结果均充分证明了该方法卓越的性能。

关键词: 对手建模, 多智能体博弈对抗, 扩散模型, 交叉注意力机制, 在线微调

Abstract: As a key technology in multi-agent game confrontation, opponent modeling aims to learn the behavior of the opponent to reduce the uncertainty of the environment and facilitate decision-making. However, most existing opponent modeling methods adopt the structure of offline training and online adaptation. During offline training, traditional neural dynamics model is used to predict the agent step-by-step, which facilitates the formation of single-step and cumulative errors. Additionally, when facing an unknown opponent during online adaptation, the planned states of the controlled agent deviate from the distribution of the dataset. To solve these problems, a method based on a diffusion model and cross-attention is proposed to establish a correlation with the opponent. The cumulative bias problem is solved using the feature that the diffusion model can generate multistep planning sequences simultaneously. The concept of a strategy set is proposed, and the deviation problem is not only solved by online fine-tuning but also prevents the problem that the offline strategy will be destroyed in the initial stages of online training. Experimental results in both open intensive reward and sparse reward competitive environments demonstrate the superior performance of this method.

Key words: opponent modeling, multi-agent game confrontation, diffusion model, cross-attention mechanism, online fine-tuning

中图分类号:

TP391.9

何明研, 李斯源, 刘鹏, 黄剑华. 基于交叉注意力扩散模型的对手建模[J]. 计算机工程, 2026, 52(7): 76-85.

HE Mingyan, LI Siyuan, LIU Peng, HUANG Jianhua. Opponent Modeling Based on Cross-Attention Diffusion Model[J]. Computer Engineering, 2026, 52(7): 76-85.

https://www.ecice06.com/CN/Y2026/V52/I7/76

参考文献

[1] FOERSTER J N, CHEN R Y, AL-SHEDIVAT M, et al. Learning with opponent-learning awareness[EB/OL].[2024-11-14]. https://arxiv.org/abs/1709.04326.
[2] HE H, BOYD-GRABER J, KWOK K, et al. Opponent modeling in deep reinforcement learning[EB/OL].[2024-11-14]. https://arxiv.org/abs/1609.05559.
[3] KIM D K, LIU M, RIEMER M D, et al. A policy gradient algorithm for learning to learn in multiagent reinforcement learning[C]//Proceedings of International Conference on Machine Learning.[S. l.]: PMLR, 2021: 5541-5550.
[4] 王腾, 黄俊松, 王乐庭, 等. 基于MADDPG的多阵面相控阵雷达引导搜索资源优化算法[J]. 计算机工程, 2024, 50(11): 38-48. WANG T, HUANG J S, WANG L T, et al. Multi-antenna phased array radar-guided search resource optimization algorithm based on MADDPG[J]. Computer Engineering, 2024, 50(11): 38-48. (in Chinese)
[5] 施伟, 冯旸赫, 程光权, 等. 基于深度强化学习的多机协同空战方法研究[J]. 自动化学报, 2021, 47(7): 1610-1623. SHI W, FENG Y H, CHENG G Q, et al. Research on multi-aircraft cooperative air combat method based on deep reinforcement learning[J]. Acta Automatica Sinica, 2021, 47(7): 1610-1623. (in Chinese)
[6] JING Y, LI K, LIU B, et al. Towards offline opponent modeling with in-context learning[C]//Proceedings of the 12th International Conference on Learning Representations. New York, USA: ACM Press, 2023: 1-13.
[7] 白天, 吕璐瑶, 李储, 等. 基于深度强化学习的游戏智能引导算法[J]. 吉林大学学报(理学版), 2025, 63(1): 91-98. BAI T, Lü L Y, LI C, et al. Game intelligent guidance algorithm based on deep reinforcement learning[J]. Journal of Jilin University (Science Edition), 2025, 63(1): 91-98. (in Chinese)
[8] JIANG H B, JIANG J C, LU Z Q, et al. Model-based opponent modeling[C]//Proceedings of the Advances in Neural Information Processing Systems. New Orleans, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2022: 28208-28221.
[9] 徐浩添, 秦龙, 曾俊杰, 等. 基于深度强化学习的对手建模方法研究综述[J]. 系统仿真学报, 2023, 35(4): 671-694. XU H T, QIN L, ZENG J J, et al. Research progress of opponent modeling based on deep reinforcement learning[J]. Journal of System Simulation, 2023, 35(4): 671-694. (in Chinese)
[10] LI S, ZHAO H D. A survey on representation learning for user modeling[C]//Proceedings of the 29th International Joint Conference on Artificial Intelligence. Yokohama, Japan: International Joint Conferences on Artificial Intelligence Organization, 2020: 4997-5003.
[11] ROSMAN B, HAWASLY M, RAMAMOORTHY S. Bayesian policy reuse[J]. Machine Learning, 2016, 104(1): 99-127.
[12] HERNANDEZ-LEAL P, TAYLOR M E, ROSMAN B, et al. Identifying and tracking switching, non-stationary opponents: a Bayesian approach[C]//Proceedings of the AAAI Workshop on Multiagent Interaction without Prior Coordination. Palo Alto, USA: AAAI Press, 2016: 1-15.
[13] ZHENG Y, MENG Z P, HAO J Y, et al. A deep Bayesian policy reuse approach against non-stationary agents[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2018: 962-997.
[14] YANG T P, MENG Z P, HAO J Y, et al. Towards efficient detection and optimal response against sophisticated opponents[EB/OL].[2024-11-14]. https://arxiv.org/abs/1809.04240.
[15] HONG Z W, SU S Y, SHANN T Y, et al. A deep policy inference Q-network for multi-agent systems[EB/OL].[2024-11-14]. https://arxiv.org/abs/1712.07893.
[16] AL-SHEDIVAT M, BANSAL T, BURDA Y, et al. Continuous adaptation via meta-learning in nonstationary and competitive environments[EB/OL].[2024-11-14]. https://arxiv.org/abs/1710.03641.
[17] SOHL-DICKSTEIN J, WEISS E A, MAHESWARANATHAN N, et al. Deep unsupervised learning using nonequilibrium thermodynamics[EB/OL].[2024-11-14]. https://arxiv.org/abs/1503.03585.
[18] ROMBACH R, BLATTMANN A, LORENZ D, et al. High-resolution image synthesis with latent diffusion models[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 10674-10685.
[19] KONG Z F, PING W, HUANG J J, et al. DiffWave: a versatile diffusion model for audio synthesis[EB/OL].[2024-11-14]. https://arxiv.org/abs/2009.09761.
[20] 陈子民, 关志涛. 基于条件扩散模型的图像分类对抗样本防御方法[J]. 计算机工程, 2024, 50(12): 296-305. CHEN Z M, GUAN Z T. Image classification adversarial example defense method based on conditional diffusion model[J]. Computer Engineering, 2024, 50(12): 296-305. (in Chinese)
[21] JANNER M, DU Y L, TENENBAUM J B, et al. Planning with diffusion for flexible behavior synthesis[EB/OL].[2024-11-14]. https://arxiv.org/abs/2205.09991.
[22] WANG Z D, HUNT J J, ZHOU M Y. Diffusion policies as an expressive policy class for offline reinforcement learning[EB/OL].[2024-11-14]. https://arxiv.org/abs/2208.06193.
[23] LU C, BALL P, TEH Y W, et al. Synthetic experience replay[EB/OL].[2024-11-14]. https://arxiv.org/abs/2303.06614.
[24] XIAN Z, GKANATSIOS N, GERVET T, et al. ChainedDiffuser: unifying trajectory diffusion and keypose prediction for robotic manipulation[EB/OL].[2024-11-14]. https://www.semanticscholar.org/paper/ChainedDiffuser%3A-Unifying-Trajectory-Diffusion-and-Xian-Gkanatsios/c36f3635e090aba84e5e83b904a7697e83730be6.
[25] BAI C J, HE H R, LI X L, et al. Diffusion model is an effective planner and data synthesizer for multi-task reinforcement learning[C]//Proceedings of the Advances in Neural Information Processing Systems. New Orleans, USA: Neural Information Processing Systems Foundation, Inc. (NeurIPS), 2023: 64896-64917.
[26] ZHANG M Y, CAI Z A, PAN L, et al. MotionDiffuse: text-driven human motion generation with diffusion model[EB/OL].[2024-11-14]. https://arxiv.org/abs/2208.15001.
[27] AJAY A, DU Y L, GUPTA A, et al. Is conditional generative modeling all you need for decision-making?[EB/OL].[2024-11-14]. https://arxiv.org/abs/2211.15657.
[28] YANG Y, WANG J. An overview of multi-agent reinforcement learning from game theoretical perspective[EB/OL].[2024-11-14]. https://arxiv.org/pdf/2011.00583v3.
[29] ZHENG Q, ZHANG A, GROVER A. Online decision Transformer[C]//Proceedings of the International Conference on Machine Learning.[S. l.]: PMLR, 2022: 27042-27059.
[30] HUSSEIN A, GABER M M, ELYAN E, et al. Imitation learning: a survey of learning methods[J]. ACM Computing Surveys, 2018, 50(2): 1-35.
[31] AGRAWAL P, NAIR A, ABBEEL P, et al. Learning to poke by poking: experiential learning of intuitive physics[EB/OL].[2024-11-14]. https://arxiv.org/abs/1606.07419.
[32] LANCTOT M, LOCKHART E, LESPIAU J B, et al. OpenSpiel: a framework for reinforcement learning in games[EB/OL].[2024-11-14]. https://arxiv.org/abs/1908.09453.
[33] LOWE R, WU Y, TAMAR A, et al. Multi-agent actor-critic for mixed cooperative-competitive environments[EB/OL].[2024-11-14]. https://arxiv.org/abs/1706.02275.
[34] PAPOUDAKIS G, CHRISTIANOS F, ALBRECHT S V. Agent modelling under partial observability for deep reinforcement learning[C]//Proceedings of the International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2021: 19210-19222.
[35] ZINTGRAF L, DEVLIN S, CIOSEK K, et al. Deep interactive Bayesian reinforcement learning via meta-learning[EB/OL].[2024-11-14]. https://arxiv.org/abs/2101.03864.
[36] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[EB/OL].[2024-11-14]. https://arxiv.org/abs/1707.06347.
[37] PRAJAPAT M, AZIZZADENESHELI K, LINIGER A, et al. Competitive policy optimization[EB/OL].[2024-11-14]. https://arxiv.org/abs/2006.10611.

选择文件类型/文献管理软件名称

选择包含的内容