摘要： 小基站的密集随机部署会产生严重干扰和较高能耗问题，为降低网络干扰、保证用户网络服务质量（QoS）并提高网络能效，构建一种基于深度强化学习（DRL）的资源分配和功率控制联合优化框架。综合考虑超密集异构网络中的同层干扰和跨层干扰，提出对频谱与功率资源联合控制能效以及用户QoS的联合优化问题。针对该联合优化问题的NP-Hard特性，提出基于DRL框架的资源分配和功率控制联合优化算法，并定义联合频谱和功率分配的状态、动作以及回报函数。利用强化学习、在线学习和深度神经网络线下训练对网络资源进行控制，从而找到最佳资源和功率控制策略。仿真结果表明，与枚举算法、Q-学习算法和两阶段算法相比，该算法可在保证用户QoS的同时有效提升网络能效。

关键词: 超密集网络, 能效, 资源分配, 强化学习, 功率控制, 深度学习

Abstract: Dense random deployment of small base stations will cause serious interferences and significant energy consumption problems.In order to reduce network interference,ensure users' network Quality of Service(QoS) and improve network Energy Efficiency(EE),this paper constructs a joint optimization framework based on Deep Reinforcement Learning(DRL) for resource allocation and power control.By comprehensively considering the same layer interference and cross layer interference in ultra-dense heterogeneous networks,the joint optimization of energy efficiency and user QoS for joint control of spectrum and power resources is proposed.To address the NP-hard characteristics of the joint optimization problem,a joint optimization algorithm based on DRL framework for resource allocation and power control is proposed,and the states,actions and return functions of joint spectrum and power allocation are defined.Then reinforcement learning,online learning,and offline training of deep neural network are used to control network resources,so as to find the best resource and power control strategy.Simulation results show that compared with the enumeration algorithm,Q-learning algorithm and two-stage algorithm,the proposed algorithm can effectively improve network energy efficiency while ensuring users' QoS.

Key words: ultra-dense network, Energy Efficiency(EE), resource allocation, Reinforcement Learning(RL), power control, deep learning

中图分类号: 

• TN929.5