基于启发式强化学习的动态CRE偏置选择算法

doi:10.19678/j.issn.1000-3428.0055183

摘要/Abstract

摘要： 随着通信用户数量的不断增长，低功率基站逐渐出现负载不均衡问题，小区边缘用户受到的干扰逐步增加，从而导致整个小区的通信质量降低。为解决该问题，针对双层异构网络场景，提出一种基于启发函数进行小区范围扩展（CRE）偏置值动态选择的HSARSA（λ）算法。利用启发函数改进强化学习中的SARSA（λ）算法，通过该算法寻找出最优CRE偏置值，以缓解宏基站高热点负载压力并提高网络容量。仿真结果表明，相比SARSA（λ）和Q-Learning算法，HSARSA（λ）算法的边缘用户吞吐量分别提高约7%和12%，系统能效分别提高约11%与13%，系统通信质量得到较大提升。

关键词: 小区范围扩展, 负载均衡, 强化学习, SARSA(λ)算法, 能效

Abstract: With the number of communication users increasing,the load of low-power base stations gets unbalanced,resulting in the gradually rising interference of cell edge users followed by reduced communication quality of the whole cell.To address the problem,this paper proposes a HSARSA(λ) algorithm based on heuristic function for dynamic selection of Cell Range Extension (CRE) bias value in dual-layer heterogeneous networks.The heuristic function is used to improve the SARSA(λ) algorithm in Reinforcement Learning(RL),and the algorithm is adopted to find out the optimal CRE bias value,so as to relieve the high hot spot load pressure of the macro base station and improve the network capacity.Simulation results show that compared with the SARSA(λ) and Q-Learning algorithms,the throughput of edge users of the system obtained by the proposed algorithm is improved by 7% and 12% respectively,and the energy efficiency of the system is improved by 11% and 13%,which indicates a significant increase in the communication quality of the system.

Key words: Cell Range Expansion(CRE), load balancing, Reinforcement Learning(RL), SARSA(λ) algorithm, energy efficiency

中图分类号:

TP391

谷静, 邓逸飞, 张新. 基于启发式强化学习的动态CRE偏置选择算法[J]. 计算机工程, 2020, 46(5): 200-206.

GU Jing, DENG Yifei, ZHANG Xin. Dynamic CRE Bias Selection Algorithm Based on Heuristic Reinforcement Learning[J]. Computer Engineering, 2020, 46(5): 200-206.

http://www.ecice06.com/CN/Y2020/V46/I5/200

图/表 9

20200513204200

20200513204205

20200513204209

20200513204213

20200513204217

20200513204221

20200513204226

20200513204230

20200513204236

参考文献

[1] YOU Xiaohu,PAN Zhiwen,GAO Xiqi,et al.The 5G mobile communication:the development trends and its emerging key techniques[J].Science in China(Information Sciences),2014,44(5):551-563.(in Chinese)尤肖虎,潘志文,高西奇,等.5G移动通信发展趋势与若干关键技术[J].中国科学:信息科学,2014,44(5):551-563.
[2] NTT D.Evaluation scenarios and assumptions for intra-eNB COMP[C]//Proceedings of 3GPP Tsg Ran Wg1 Meeting.Valencia,Spain:[s.n.],2010:25-33.
[3] TARIQ F,DOOLEY L,ALLEN B,et al.A cell range expansion framework for closed access Femtocell networks[J].Wireless Personal Communications,2017,93(3):601-614.
[4] ZHANG L,ZHAO S J,SHANG P,et al.Distributed adaptive range extension setting for small cells in heterogeneous cellular network[C]//Proceedings of 2017 IEEE Vehicular Technology Conference.Washington D.C.,USA:IEEE Press,2017:1-7.
[5] DHUNGANA Y,TELLAMBURA C.Multichannel analysis of cell range expansion and resource partitioning in two-tier heterogeneous cellular networks[J].IEEE Transactions on Wireless Communications,2016,15(3):2394-2406.
[6] ZHANG Y,CUI Z Y,CUI Q M,et al.Energy efficiency analysis of heterogeneous cellular networks with extra cell range expansion[J].IEEE Access,2017,5:11003-11014.
[7] HU H N,WENG J L,ZHANG J.Coverage performance analysis of FeICIC low-power subframes[J].IEEE Transactions on Wireless Communications,2016,15(8):5603-5614.
[8] BALACHANDRAN K,KANG J H,KARAKAYALI K,et al.Virtual soft-handoff for cellular heterogeneous networks[J].IEEE Transactions on Vehicular Technology,2017,66(4):3306-3318.
[9] AYALA-ROMERO J A,ALCARAZ J J,VALES-ALONSO J.Data-driven configuration of interference coordination parameters in HetNets[J].IEEE Transactions on Vehicular Technology,2018,67(6):5174-5187.
[10] LI Yixiao.Research on hybrid access performance analysis and admission control mechanism in Femtocell[D].Chongqing:Chongqing University of Posts and Telecommunications,2013.(in Chinese)李毅潇.Femtocell中混合接入性能分析及接纳控制机制研究[D].重庆:重庆邮电大学,2013.
[11] XIAO Zhu,LI Renfa,YI Kechu,et al.Development and prospect of research on femtocell in two-tier heterogeneous networks[J].Journal on Communications,2013,34(2):156-169.(in Chinese)肖竹,李仁发,易克初,等.两层异构网络中femtocell研究进展与展望[J].通信学报,2013,34(2):156-169.
[12] LOPEZ-PEREZ D,CHU X L.Inter-cell interference coordination for expanded region picocells in heterogeneous networks[C]//Proceedings of the 20th International Conference on Computer Communications and Networks.Washington D.C.,USA:IEEE Press,2011:15-33.
[13] SUN Y Z,XIA W Q,ZHANG S Q,et al.Energy efficient pico cell range expansion and density joint optimization for heterogeneous networks with eICIC[J].Sensors,2018,18(3):762-765.
[14] HOU J,LI H,HU J,et al.A review of the applications and hotspots of reinforcement learning[C]//Proceedings of 2017 IEEE International Conference on Unmanned Systems.Washington D.C.,USA:IEEE Press,2017:56-63.
[15] SUTTONR S,BARTOA G.Reinforcement learning[M].Cambridge,USA:MIT Press,2017.
[16] LENG J S,JAIN L,FYFE C.Convergence analysis on approximate reinforcement learning[C]//Proceedings of International Conference on Knowledge Science,Engineering and Management.Berlin,Germany:Springer,2007:263-276.
[17] ZHANG Fang.Research on reinforcement learning method for multi mobile robot system[D].Shanghai:Shanghai Jiaotong University,2002.(in Chinese)张芳.面向多移动机器人系统的再励学习方法研究[D].上海:上海交通大学,2002.
[18] BIANCHI R A C,RIBEIRO C H C,COSTA A H R.Heuristically accelerated Q-Learning:a new approach to speed up reinforcement learning[C]//Proceedings of SBIA'04.Berlin,Germany:Springer,2004:245-254.
[19] SHIRAKABE M,MORIMOTO A,MIKI N.Performance evaluation of inter-cell interference coordination and cell range expansion in heterogeneous networks for LTE-advanced downlink[C]//Proceedings of 2011 International Symposium on Wireless Communication Systems.Washington D.C.,USA:IEEE Press,2011:56-98.
[20] GALINDO-SERRANO A,GIUPPONI L,BLASCO P,et al.Learning from experts in cognitive radio networks:the docitive paradigm[C]//Proceedings of the 5th International ICST Conference on Cognitive Radio Oriented Wireless Networks and Communications.Washington D.C.,USA:IEEE Press,2010:15-26.

选择文件类型/文献管理软件名称

选择包含的内容