摘要： 在多约束复杂环境下，多数无人飞行器（UAV）航迹规划方法无法从历史经验中获得先验知识，导致对多变的环境适应性较差。提出一种基于深度强化学习的航迹规划策略自学习方法，利用飞行约束条件设计UAV的状态及动作模式，从搜索宽度和深度2个方面降低航迹规划搜索规模，基于航迹优化目标设计奖惩函数，利用由卷积神经网络引导的蒙特卡洛树搜索（MCTS）算法学习得到航迹规划策略。仿真结果表明，该方法自学习得到的航迹规划策略具有泛化能力，相对未迭代训练的网络，该策略仅需17%的NN-MCTS仿真次数就可引导UAV在未知飞行环境中满足约束条件并安全无碰撞地到达目的地。

关键词: 深度强化学习, 蒙特卡洛树搜索, 航迹规划策略, 策略自学习, 多约束, 复杂环境

Abstract: In a complex multi-constrained environment,the Unmanned Aerial Vehicle(UAV) track planning methods generally fail to obtain priori knowledge from historical experience,resulting in poor adaptability to a variable environment.To address the problem,this paper proposes a self-learning method for track planning strategy based on deep reinforcement learning.Based on the UAV flight constraints,the design of the UAV state and action modes is optimized to reduce the width and depth of track planning search.The reward and punishment function is designed based on the track optimization objective.Then,a Monte Carlo Tree Search(MCTS) algorithm guided by a convolutional neural network is used to learn the track planning strategy.Simulation results show that the track planning strategy obtained by the proposed self-learning method has generalization ability.Compared with the networks without iterative training,the strategy obtained by this method requires only 17% of the number of NN-MCTS simulation times to guide the UAV to reach the destination safely without collision and satisfy the constraints in an unknown environment.

Key words: deep reinforcement learning, Monte Carlo Tree Search(MCTS), track planning strategy, strategy self-learning, multiple constraints, complex environment

中图分类号: 

• TP242.6