A Network Access Control Algorithm Based on Q-learning

doi:10.3969/j.issn.1000-3428.2015.10.019

Abstract

Abstract: The next generation wireless network is the heterogeneous network of coexistence of a variety of wireless access technology.In order to make full use of the resources of all kinds of wireless network,the integration of heterogeneous network is necessary.However,when it comes to the heterogeneous network integration,the call request access control problem comes.In wireless heterogeneous network composing of Long Term Evolution(LTE),Wireless Local Area Network(WLAN) and Device-to-Device(D2D),an algorithm is presented for heterogeneous wireless network selection.The proposed algorithm based on Q-learning can select the appropriate network for access according to different traffic types,terminal mobility and network load status by using the return function composing of matching coefficient reflecting the network contribution,the corresponding traffic and mobility.Simulation results show that the proposed algorithm has an efficient learning ability to achieve autonomous radio resource management,which effectively improves the spectrum utility and reduces the blocking probability.

Key words: heterogeneous wireless network, access control, Long Term Evolution(LTE), Device-to-Device(D2D), Q-learning, resource management, block probability, spectrum utility

摘要： 下一代无线网络是多种无线接入技术共存的异构网络,要充分利用各种无线网络的资源,需要实现异构网络的融合,但网络融合却面临异构网络环境下的接入请求控制问题。为此,在长期演进、无线局域网、设备直连构成的无线异构网络下,提出一种异构网络接入控制算法。根据不同的业务类型、终端移动性及网络负载状态,利用匹配系数所构成的回报函数,反映网络对相应业务及移动性的贡献程度,通过Q学习算法选择合适的网络接入方式。仿真结果表明,该算法具有高效的在线学习能力,能够有效提升网络在频谱效用和阻塞率方面的性能,实现自主的无线资源管理。

关键词: 异构无线网络, 接入控制, 长期演进, 设备直连, Q学习, 资源管理, 阻塞率, 频谱效用

CLC Number:

TP393

FENG Chenwei,ZHANG Lin. A Network Access Control Algorithm Based on Q-learning[J]. Computer Engineering, doi: 10.3969/j.issn.1000-3428.2015.10.019.

冯陈伟,张璘. 一种基于Q学习的网络接入控制算法[J]. 计算机工程, doi: 10.3969/j.issn.1000-3428.2015.10.019.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.3969/j.issn.1000-3428.2015.10.019

http://www.ecice06.com/EN/Y2015/V41/I10/99

References

参考文献［1］Luo Jijun,Mukerjee R,Dillinger M,et al.Investigation of Radio Resource Scheduling in WLANs Coupled with 3G Cellular Network［J］.IEEE Communications Magazine,2003,41(6):108-115. ［2］Song Qingyang,Jamalipour A.Network Selection in an Integrated Wireless LAN and UMTS Environment Using Mathematical Modeling and Computing Techniques［J］.IEEE Wireless Communications,2005,12(3):42-48. ［3］Barto A G.Reinforcement Learning:An Introduction［M］.Cambridge,USA:MIT press,1998. ［4］Kaelbling L P,Littman M L,Moore A W.Reinforce-mentLearning:A Survey［J］.Journal of Artificial Intelligence Research,1996,9(1):99-137. ［5］Nie Junhong,Haykin S.A Q-learning-based Dynamic Channel Assignment Technique for Mobile Communica-tion Systems［J］.IEEE Transactions on Vehicular Tech-nology,1999,48(5):1676-1687. ［6］Senouci S M,Beylot A L,Pujolle G.CallAdmission Control in Cellular Networks:A Reinforcement Learning Solution［J］.International Journal of Network Manage-ment,2004,14(2):89-103. ［7］Haddad M,Altman Z,Elayoubi S E,et al.A Nash-stackelberg Fuzzy Q-learning Decision Approach in Heterogeneous Cognitive Networks［C］//Proceedings of Global Telecommunications Conference.Washington D.C.,USA:IEEE Press,2010:1-6. ［8］Simsek M,Czylwik A.Decentralized Q-learning of LTE-femtocells for Interference Reduction in Heterogeneous Networks Using Cooperation［C］//Proceedings of International ITG Workshop on Smart Antennas.Washington D.C.,USA:IEEE Press,2012:86-91. ［9］Saker L,Ben Jemaa S,Elayoubi S E.Q-learning for Joint Access Decision in Heterogeneous Networks［C］//Proceedings of Wireless Communications and Network-ing Conference.Washington D.C.,USA:IEEE Press,2009:1-5. ［10］Tabrizi H,Farhadi G,Cioffi J.DynamicHandoff Decision in Heterogeneous Wireless Systems:Q-learning Approach［C］//Proceedings of IEEE International Con-ference on Communications.Washington D.C.,USA:IEEE Press,2012:3217-3222. ［11］张永靖,冯志勇,张平.基于Q学习的自主联合无线资源管理算法［J］.电子与信息学报,2008,30(3):676-680. ［12］Zhao Yanqing,Zhou Weifeng,Zhu Qi.Q-learning Based Heterogeneous Network Selection Algorithm［J］.Journal of Computer Applications,2012,127(2):471-477. ［13］Tan Xu,Luan Xi,Cheng Yuxin,et al.Cell Selection in Two-tier Femtocell Networks Using Q-learning Algorithm［C］//Proceedings of the 16th International Conference on Advanced Communication Technology.Washington D.C.,USA:IEEE Press,2014:1031-1035. ［14］Doppler K,Rinne M,Wijting C,et al.Device-to-Device Communication as an Underlay to LTE-advanced Networks［J］.IEEE Communications Magazine,2009,47(12):42-49. ［15］Barto A G,Bradtke S J,Singh S P.Learning to Act Using Real-time Dynamic Programming［J］.Artificial Intelligence,1995,72(1):81-138. ［16］Watkins C J C H,Dayan P.Q-learning［J］.Machine-Learning,1992,8(3/4):279-292. ［17］叶培智.异构无线网络接入选择算法的研究［D］.厦门:厦门大学,2013. 编辑顾逸斐

[1]	Baohua HUANG, Huiying ZHENG, Xi QU, Ningjiang CHEN. Efficient Storage Access Control Scheme for Alliance Chain [J]. Computer Engineering, 2023, 49(8): 37-45.
[2]	ZHANG Zundong, WANG Yannan, ZHOU Huijuan, ZHANG Yifan. The Influence of Decision Mechanisms on Network Cooperation Level in Q-learning Evolutionary Game [J]. Computer Engineering, 2023, 49(6): 99-106,114.
[3]	ZHANG Guofu, SHEN Yufeng, SONG Xiaoxiao, SU Zhaopin. Method for Modeling and Solving the Dynamic Scheduling Problem of the Repair Crew for Restoring Damaged Road Network [J]. Computer Engineering, 2023, 49(6): 300-313.
[4]	BI Xiang, HUANG Huang, ZHANG Benhong, WEI Xing. V2V Composite Routing Algorithm for Internet of Vehicles Based on Clustering and Improved Q-Learning [J]. Computer Engineering, 2023, 49(3): 221-230,247.
[5]	TIAN Xiuxia, YANG Mingyi. Smart Contract-Based Access Control Mechanism in Home IoT [J]. Computer Engineering, 2023, 49(3): 18-28.
[6]	ZHAO Beiying, JI Weifeng, WENG Jiang, WU Xuan, LI Yingqi. Trusted Routing Algorithm Based on Heuristic Q-Learning for FANET [J]. Computer Engineering, 2022, 48(5): 162-169.
[7]	LIU Jing, ZHU Bingxu, LIANG Jiahang, REN Nüer, JI Haipeng. Blockchain Access Control Strategy Based on Mainchain and Sidechain Cooperation [J]. Computer Engineering, 2022, 48(3): 10-16,22.
[8]	ZHANG Ran, GAO Yingxue, ZHAO Yu, DING Yuanming. Routing Algorithm for Micro-Nano-Satellite Based on Q-Learning Quantum Ant Colony [J]. Computer Engineering, 2022, 48(3): 162-169,188.
[9]	WANG Jingyu, ZHOU Xuejuan. An Attribute-based Encryption Scheme for Ciphertext Policy that Supports Attribute Revocation [J]. Computer Engineering, 2021, 47(7): 95-100.
[10]	ZHANG Benhong, WU Haohao, YU Lei. Contention-based Time Division Multiple Access MAC Protocol in Vehicular Ad Hoc Networks [J]. Computer Engineering, 2021, 47(5): 154-159.
[11]	ZHANG Jianguo, HU Xiaohui. Research on Improved Access Control Mechanism of Internet of Things Devices Based on Ethereum [J]. Computer Engineering, 2021, 47(4): 32-39,47.
[12]	JIANG Baoqing, CHEN Hongbin. Trajectory Planning for Unmanned Aerial Vehicle Assisted WSN Data Collection Based on Q-Learning [J]. Computer Engineering, 2021, 47(4): 127-134,165.
[13]	ZHANG Jianghui, CUI Bo, LI Ru, SHI Jinshan. Access Control System of Internet of Things Based on Smart Contract [J]. Computer Engineering, 2021, 47(4): 21-31.
[14]	ZHAN Changjian, LEI Lei, SHEN Gaoqing, LI Zhilin. Multi-Link Transmission MAC Protocol Suitable for Underwater Acoustic Sensor Network [J]. Computer Engineering, 2021, 47(2): 194-200.
[15]	ZHOU Yunteng, ZHANG Xueying, LI Fenglian, LIU Shuchang, JIAO Jiangli, TIAN Dou. SVDPP Recommendation Algorithm Optimized by Q-learning Algorithm [J]. Computer Engineering, 2021, 47(2): 46-51.

Please choose a citation manager

Content to export