基于深度强化学习的人机共享控制方法

doi:10.19678/j.issn.1000-3428.0260592

计算机工程

• •

基于深度强化学习的人机共享控制方法

针对现有人机共享控制策略在复杂交互场景下难以同时兼顾安全性与人机友好性的问题，提出一种融合驾驶员意图与孪生价值分布的人机共享控制方法。首先，构建了基于双层门控循环单元网络（GRU）的驾驶员意图预测模型，通过提取驾驶员控制指令的时序关联特征，实现准确的驾驶员意图在线预测；其次，针对复杂交互场景下的“人-车-环境”强耦合特性，建立了面向人机共享控制的马尔可夫决策过程（MDP），将环境状态、车辆状态和驾驶员状态统一建模为多维增强状态空间，并设计了兼顾驾驶员意图跟踪以及行驶安全的奖励函数，从而通过闭环交互学习兼顾安全性与人机友好性的共享控制策略；再次，针对策略Q值高估导致策略次优及非必要干预问题，提出了基于孪生价值分布的动作价值评估方法，将动作价值建模为概率分布，通过保守价值估计抑制策略学习中的Q值高估问题，并利用动作价值标准差刻画价值估计不确定性，避免策略次优并提升共享控制策略的友好性。最后，通过仿真实验与人在环实验验证所提方法在复杂交互场景中的有效性，实验结果表明，所提方法任务成功率和人机友好性均优于软演员-评论家算法（SAC）、近端策略优化算法（PPO）和深度确定性策略梯度算法（DDPG）等代表性的深度强化学习方法，以及近年来提出的分布式强化学习和人在环强化学习相关方法，实现了安全友好的人机共享控制。

发布日期:2026-11-04

Human-Machine Shared Control Method Based on Deep Reinforcement Learning

To address the difficulty of balancing safety and human-machine friendliness in complex interactive scenarios, this paper proposes a human-machine shared control method that integrates driver intention and twin value distributions. First, this paper constructs a driver intention prediction model based on a two-layer gated recurrent unit(GRU) network. The model extracts temporal correlation features from driver control commands and predicts driver intention online. Second, this paper establishes a Markov decision process(MDP) for human-machine shared control according to the strong coupling characteristics of the human-vehicle-environment system in complex interactive scenarios. It models environmental states, vehicle states, and driver states as a unified multidimensional enhanced state space. It also designs a reward function that considers both driver intention tracking and driving safety. This design enables the agent to learn a shared control policy with both safety and human-machine friendliness through closed-loop interaction. Third, this paper proposes an action-value evaluation method based on twin value distributions to address the suboptimal policy and unnecessary intervention problems caused by Q-value overestimation. The method models the action value as a probability distribution. It suppresses Q-value overestimation in policy learning through conservative value estimation. It also uses the standard deviation of the action-value distribution to characterize the stability of value estimation, avoids suboptimal policies, and improves the friendliness of the shared control policy. Finally, simulation experiments and human-in-the-loop experiments verify the effectiveness of the proposed method in complex interactive scenarios. The experimental results show that the proposed method outperforms representative deep reinforcement learning methods, including Soft Actor-Critic(SAC), Proximal Policy Optimization(PPO), and Deep Deterministic Policy Gradient(DDPG), as well as recently proposed methods related to distributional reinforcement learning and human-in-the-loop reinforcement learning, in terms of task success rate and human-machine friendliness. The proposed method achieves safe and friendly human-machine shared control.

Published:2026-11-04

摘要/Abstract

王自超, 黎谦, 秦业菲, 梁洋, 尹智帅, 聂琳真. 基于深度强化学习的人机共享控制方法[J]. 计算机工程, doi: 10.19678/j.issn.1000-3428.0260592.

WANG Zichao, LI Qian, QIN Yefei, LIANG Yang, YIN Zhishuai, NIE Linzhen. Human-Machine Shared Control Method Based on Deep Reinforcement Learning[J]. Computer Engineering, doi: 10.19678/j.issn.1000-3428.0260592.

参考文献

[1] WANG W S, NA X X, CAO D P, et al. Decision-making in driver-automation shared control: A review and perspectives[J]. IEEE/CAA Journal of Automatica Sinica, 2020, 7(5): 1289-1307.
[2] HUANG C, LV C, HANG P, et al. Human-machine adaptive shared control for safe driving under automation degradation[J]. IEEE Intelligent Transportation Systems Magazine, 2021, 14(2): 53-66.
[3] HUANG T, FU R, SUN Q Y, et al. Toward human-vehicle collaboration for automated vehicles: A review and perspective[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(5): 5649-5673.
[4] ZHANG Y, LI Z H, HU C, et al. Human-machine shared steering decision-making of intelligent vehicles based on heterogeneous synchronous reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2025, 26(9): 13869-13881.
[5] 杨俊儒, 褚端峰, 陆丽萍, 等. 智能汽车人机共享控制研究综述[J]. 机械工程学报, 2022, 58(18): 31-55. YANG J R, CHU D F, LU L P, et al. Review on Human-machine Shared Control of Intelligent Vehicles[J]. Journal of Mechanical Engineering, 2022, 58(18): 31-55.
[6] ABBINK D A, Mulder M. Neuromuscular analysis as a guideline in designing shared control[M]// HOSSEINI M. Advances in haptics. Wenen, Oostenrijk: Intech, 2010: 499-516.
[7] MARS F, DEROO M, HOC J M. Analysis of human-machine cooperation when driving with different degrees of haptic shared control[J]. IEEE transactions on haptics, 2014, 7(3): 324-333.
[8] ZHAO X B, YIN Z S, HE Z W, et al. Indirect Shared Control Strategy for Human-Machine Cooperative Driving on Hazardous Curvy Roads[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(3): 2257-2270.
[9] TRAN D, DU J H, SHENG W H, et al. A human-vehicle collaborative driving framework for driver assistance[J]. IEEE Transactions on Intelligent Transportation Systems, 2018, 20(9): 3470-3485.
[10] SHI Z Q, CHEN H, QU Q, et al. Human–Machine Cooperative Steering Control Considering Mitigating Human–Machine Conflict Based on Driver Trust [J]. IEEE Transactions on Human-Machine Systems, 2022, 52(5): 1036-1048.
[11] Li M, SONG X, Cao H, et al. Shared control with a novel dynamic authority allocation strategy based on game theory and driving safety field[J]. Mechanical Systems and Signal Processing, 2019, 124: 199-216.
[12] NA X X, COLE D J. Application of open-loop stackelberg equilibrium to modeling a driver's interaction with vehicle active steering control in obstacle avoidance[J]. IEEE Transactions on Human-Machine Systems, 2017, 47(5): 673-685.
[13] LIANG Y, YIN Z S, NIE L Z , et al. Shared steering control with predictive risk field enabled by digital twin[J]. IEEE Transactions on Intelligent Vehicles, 2023, 8(5): 3256-3269.
[14] ERLIEN S M, FUJITA S, GERDES J C. Shared steering control using safe envelopes for obstacle avoidance and vehicle stability[J]. IEEE Transactions on Intelligent Transportation Systems, 2015, 17(2): 441-451.
[15] 张祥瑞, 谭泰, 李辉, 等. 基于深度强化学习的无人机空战机动决策方法[J]. 计算机工程, 2025, 51(12): 324-336. ZHANG X R, TAN T, LI H, et al. Aerial Combat Maneuver Decision Method for Unmanned Aerial Vehicles Based on Deep Reinforcement Learning[J]. Computer Engineering, 2025, 51(12): 324-336.
[16] 叶宝林, 王欣, 李灵犀, 等. 基于深度强化学习PPO的车辆智能控制方法[J]. 计算机工程, 2025, 51(07): 385-396. YE B L, WANG X, LI L X, et al. Vehicle Intelligent Control Method Based on Deep Reinforcement Learning PPO[J]. Computer Engineering, 2025, 51(07): 385-396.
[17] XIE J, XU X, WANG F, et al. Coordination control strategy for human-machine cooperative steering of intelligent vehicles: A reinforcement learning approach[J]. IEEE Transactions on Intelligent Transportation Systems, 2022, 23(11): 21163-21177.
[18] Yan L, Wu X, Wei C, et al. Human-vehicle shared steering control for obstacle avoidance: A reference-free approach with reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(11): 17888-17901.
[19] HUANG Z Y, WU J D, LV C. Efficient deep reinforcement learning with imitative expert priors for autonomous driving[J]. IEEE transactions on neural networks and learning systems, 2022, 34(10): 7391-7403.
[20] HUANG W H, LIU H C, HUANG Z Y, et al. Safety-aware human-in-the-loop reinforcement learning with shared control for autonomous driving[J]. IEEE Transactions on Intelligent Transportation Systems, 2024, 25(11): 16181-16192.
[21] HUANG Z L, SHENG Z H, MA C Y, et al. Human as AI mentor: Enhanced human-in-the-loop reinforcement learning for safe and efficient autonomous driving[J]. Communications in Transportation Research, 2024, 4: 100127.
[22] YANG C, ZHU Y, CHEN Y. A review of human-machine cooperation in the robotics domain[J]. IEEE Transactions on Human-Machine Systems, 2022, 52(1): 12-25.
[23] VAN HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double q-learning[C]//Proceedings of the AAAI conference on artificial intelligence. Palo Alto, USA: 2016, 30(1).
[24] HAARNOJA T, ZHOU A, ABBEEL P, et al. Soft actor-critic: Off-policy maximum entropy deep reinforcement learning with a stochastic actor[C]//Proceedings of the 35th International conference on machine learning. Stockholm, Sweden: 2018: 1861-1870.
[25] FUJIMOTO S, VAN HOOF H, MEGER D. Addressing function approximation error in actor-critic methods[C]//Proceedings of the 35th International Conference on Machine Learning. Stockholm, Sweden: PMLR, 2018: 1587-1596.
[26] BELLEMARE M G, DABNEY W, MUNOS R. A distributional perspective on reinforcement learning[C]//Proceedings of the 34th International Conference on Machine Learning. Sydney, Australia: PMLR, 2017: 449-458.
[27] DUAN J L, GUAN Y, LI S E, et al. Distributional soft actor-critic: off-policy reinforcement learning for addressing value estimation errors[J]. IEEE Transactions on Neural Networks and Learning Systems, 2022, 33(11): 6584-6598.
[28] CHOW Y, TAMAR A, MANNOR S, et al. Risk-sensitive and robust decision-making: a CVaR optimization approach[C]//Advances in Neural Information Processing Systems. Montreal, Canada: Curran Associates, 2015: 1522-1530.
[29] SCHULMAN J, WOLSKI F, DHARIWAL P, et al. Proximal policy optimization algorithms[EB/OL]. [2026-06-14]. https://arxiv.org/abs/1707.06347.
[30] LILLICRAP T P, HUNT J J, PRITZEL A, et al. Continuous control with deep reinforcement learning[C]//International Conference on Learning Representations. San Juan, Puerto Rico: ICLR, 2016.
[31] DUAN J L, WANG W X, XIAO L M, et al. Distributional Soft Actor-Critic With Three Refinements[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2025, 47(5): 3935-3946.
[32] HUANG Z Y, WU J D, LV C. Driving behavior modeling using naturalistic human driving data with inverse reinforcement learning[J]. IEEE Transactions on Intelligent Transportation Systems, 2021, 23(8): 10239-10251.
[33] CAESAR H, BANKITI V, LANG A H, et al. nuScenes: A Multimodal Dataset for Autonomous Driving[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE, 2020: 11618-11628.
[34] ELMAN J L. Finding structure in time[J]. Cognitive Science, 1990, 14(2): 179-211.
[35] HOCHREITER S, SCHMIDHUBER J. Long short-term memory[J]. Neural Computation, 1997, 9(8): 1735-1780.
[36] BAI S, KOLTER J Z, KOLTUN V. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling[EB/OL]. [2026-06-14]. arXiv:1803.01271.
[37] VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Advances in Neural Information Processing Systems. Long Beach, USA: Curran Associates, 2017, 30: 5998-6008.
[38] LIU Y, HU T G, ZHANG H R, et al. iTransformer: Inverted Transformers Are Effective for Time Series Forecasting[C]//International Conference on Learning Representations. Vienna, Austria: ICLR, 2024.
[39] WANG S Y, WU H X, SHI X M, et al. TimeMixer: Decomposable Multiscale Mixing for Time Series Forecasting[C]//International Conference on Learning Representations. Vienna, Austria: ICLR, 2024.
[40] ZHOU P F, LIU Y L, LIANG J L, et al. CrossLinear: Plug-and-Play Cross-Correlation Embedding for Time Series Forecasting With Exogenous Variables[C]//Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Toronto, Canada: ACM, 2025: 4120-4131.
[41] QIU X F, WU X J, LIN Y, et al. DUET: Dual Clustering Enhanced Multivariate Time Series Forecasting[C]//Proceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining. Toronto, Canada: ACM, 2025: 1185-1196.

选择文件类型/文献管理软件名称

选择包含的内容

基于深度强化学习的人机共享控制方法

Human-Machine Shared Control Method Based on Deep Reinforcement Learning

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于深度强化学习的人机共享控制方法

Human-Machine Shared Control Method Based on Deep Reinforcement Learning

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics

本文评价