A3C-Based Interference Cancellation Algorithm for Cognitive Internet of Things Communication

doi:10.19678/j.issn.1000-3428.0068319

Abstract

Abstract:

To address the intelligent needs of spectrum resource interference management, in this study, an Asynchronous Advantage Actor-Critic(A3C)-based intelligent interference elimination algorithm is proposed to tackle the interference problem caused by spectrum resource sharing in a Cognitive Internet of Things (CIoT) communication system. Through learning and optimization of the agent, the algorithm helps Secondary Users (SU) make optimal decisions when affected by interference, thereby improving communication quality and system performance. When the communication quality of the SU is affected by interference, the learning and optimization of the agent enables the SU to choose the action with the lowest interference degree according to the current position information, transmit power, and receive power and interference degree, whereby a reward is offered after performing the action. Intelligent agents attempt different actions to reduce interference and adjust their strategies based on the reward feedback, thereby maximizing the reward function by considering the interference level and signal quality indicators and minimizing the impact of interference on communication quality. The simulation results show that compared with the traditional k-means, Deep Recurrent Q-Network (DRQN), and Deep Q-Network (DQN) optimization algorithms, the A3C-based interference cancellation algorithm has a shorter convergence time, higher execution efficiency, and at least 7% higher throughput performance than the three benchmark methods. This demonstrates that the proposed algorithm can effectively reduce the adverse effects of interference on communication quality.

Key words: Cognitive Internet of Things(CIoT), interference cancellation, Asynchronous Advantage Actor-Critic(A3C) algorithm, interference level, signal quality, throughput

摘要：

针对频谱资源干扰管理的智能化需求, 提出一种基于异步优势行动者-评论家(A3C)的干扰消除算法, 旨在应对认知物联网(CIoT)通信系统中由频谱资源共享引起的干扰问题。通过智能体的学习和优化, 帮助次级用户(SU)在受到干扰影响时做出最优的决策, 从而改善通信质量和系统性能。在该算法中, 当SU遭受干扰影响通信质量时, 智能体通过学习和优化, 使SU能够根据当前的位置信息、发射功率、接收功率以及干扰程度选择最低干扰程度的行动, 并执行该行动后获得的奖励。智能体通过尝试不同减少干扰的行动, 并根据奖励的反馈调整策略, 达到最大化定义干扰程度指标和信号质量指标的奖励函数的目的, 从而最大程度地减少干扰对通信质量的影响。实验结果表明, 与传统k-means算法以及深度递归Q网络(DRQN)和深度Q网络(DQN)优化算法相比, 基于A3C的干扰消除算法具有更短的收敛时间、更高的执行效率以及更高的系统吞吐量, 较3种基准方法在吞吐量性能上至少提高7%, 能够有效地减少干扰对通信质量的不利影响。

关键词: 认知物联网, 干扰消除, 异步优势行动者-评论家算法, 干扰程度, 信号质量, 吞吐量

LIU Xinmeng, XIE Jianli, LI Cuiran, WANG Yiming. A3C-Based Interference Cancellation Algorithm for Cognitive Internet of Things Communication[J]. Computer Engineering, 2024, 50(10): 281-290.

刘新梦, 谢健骊, 李翠然, 王亦鸣. 基于A3C的认知物联网通信干扰消除算法[J]. 计算机工程, 2024, 50(10): 281-290.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0068319

https://www.ecice06.com/EN/Y2024/V50/I10/281

Figures/Tables 12

Fig.1 Schematic diagram of CIoT communication system model

Fig.2 Overall framework for interference elimination

Fig.3 Distribution map of base stations and users

Fig.4 Convergence of algorithms in this paper under different learning rates

Fig.5 Cumulative time comparison of different algorithms

Fig.6 Relationship between interference power and system throughput

Fig.7 The relationship between system throughput and the number of SUs

Fig.8 Relationship between system throughput and SU-PU distance

Fig.9 Relationship between system throughput and SU-SU distance

References 31

1	LIU X, DING H, HU S. Uplink resource allocation for NOMA-based hybrid spectrum access in 6G-enabled cognitive Internet of Things. IEEE Internet of Things Journal, 2021, 8(20): 15049- 15058. doi: 10.1109/JIOT.2020.3007017
2	ZAHEER K, OTHMAN M, REHMANI M H, et al. A survey of decision-theoretic models for Cognitive Internet of Things (CIoT). IEEE Access, 2018, 6, 22489- 22512. doi: 10.1109/ACCESS.2018.2825282
3	WANG D Y, QI P H, FU Q F, et al. Multiple high-order cumulants-based spectrum sensing in full-duplex-enabled cognitive IoT networks. IEEE Internet of Things Journal, 2021, 8(11): 9330- 9343. doi: 10.1109/JIOT.2021.3055782
4	EJAZ W, IBNKAHLA M. Multiband spectrum sensing and resource allocation for IoT in cognitive 5G networks. IEEE Internet of Things Journal, 2018, 5(1): 150- 163. doi: 10.1109/JIOT.2017.2775959
5	LI F, LAM K Y, LI X H, et al. Advances and emerging challenges in cognitive Internet-of-Things. IEEE Transactions on Industrial Informatics, 2020, 16(8): 5489- 5496. doi: 10.1109/TII.2019.2953246
6	BALA I, AHUJA K. Energy-efficient framework for throughput enhancement of cognitive radio network by exploiting transmission mode diversity. Journal of Ambient Intelligence and Humanized Computing, 2023, 14(3): 2167- 2184. doi: 10.1007/s12652-021-03428-x
7	HE T J, CHIN K W, SOH S, et al. A novel distributed resource allocation scheme for wireless-powered cognitive radio Internet of Things networks. IEEE Internet of Things Journal, 2021, 8(20): 15486- 15499. doi: 10.1109/JIOT.2021.3071396
8	DENG J H, CHEN S H, KU M L. Multiuser MIMO precoders with proactive primary interference cancelation and link quality enhancement for cognitive radio relay systems. IEEE Access, 2017, 5, 17701- 17712. doi: 10.1109/ACCESS.2017.2749122
9	张梓扬, 常军, 黄一帆, 等. 基于强化学习的空间引力波探测望远镜系统外杂光抑制研究. 光电工程, 2024, 51(2): 71- 81.
	ZHANG Z Y, CHAGN J, HUAGN Y F, et al. Reinforcement learning-based stray light suppression study for space-based gravitational wave detection telescope system. Opto-Electronic Engineering, 2024, 51(2): 71- 81.
10	SUN C, DING H, LIU X. Multichannel spectrum access based on reinforcement learning in cognitive Internet of Things. Ad Hoc Networks, 2020, 106, 102200. doi: 10.1016/j.adhoc.2020.102200
11	TAN X, ZHOU L, WANG H J, et al. Cooperative multi-agent reinforcement-learning-based distributed dynamic spectrum access in cognitive radio networks. IEEE Internet of Things Journal, 2022, 9(19): 19477- 19488. doi: 10.1109/JIOT.2022.3168296
12	MOAYEDIAN N S, SALEHI S, KHABBAZIAN M. Fair resource allocation in cooperative cognitive radio IoT networks. IEEE Access, 2020, 8, 191067- 191079. doi: 10.1109/ACCESS.2020.3032204
13	SAFDAR MALIK T, RAZZAQ MALIK K, AFZAL A, et al. RL-IoT: reinforcement learning-based routing approach for cognitive radio-enabled IoT communications. IEEE Internet of Things Journal, 2023, 10(2): 1836- 1847. doi: 10.1109/JIOT.2022.3210703
14	XU L, YIN W X, ZHANG X L, et al. Fairness-aware throughput maximization over cognitive heterogeneous NOMA networks for industrial cognitive IoT. IEEE Transactions on Communications, 2020, 68(8): 4723- 4733. doi: 10.1109/TCOMM.2020.2992720
15	TULI S, ILAGER S, RAMAMOHANARAO K, et al. Dynamic scheduling for stochastic edge-cloud computing environments using A3C learning and residual recurrent neural networks. IEEE Transactions on Mobile Computing, 2022, 21(3): 940- 954. doi: 10.1109/TMC.2020.3017079
16	王毅然, 经小川, 贾福凯, 等. 基于多智能体协同强化学习的多目标追踪方法. 计算机工程, 2020, 46(11): 90- 96. doi: 10.3778/j.issn.1002-8331.1911-0132
	WANG Y R, JING X C, JIA F K, et al. Multi-target tracking method based on multi-agent collaborative reinforcement learning. Computer Engineering, 2020, 46(11): 90- 96. doi: 10.3778/j.issn.1002-8331.1911-0132
17	杨思明, 单征, 丁煜, 等. 深度强化学习研究综述. 计算机工程, 2021, 47(12): 19- 29. URL
	YANG S M, SHAN Z, DING Y, et al. Survey of research on deep reinforcement learning. Computer Engineering, 2021, 47(12): 19- 29. URL
18	LIU Y, YUAN X J, LIANG Y C, et al. Machine learning based iterative detection and multi-interference cancellation for cognitive IoT. IEEE Communications Letters, 2020, 24(9): 1995- 1999. doi: 10.1109/LCOMM.2020.2997048
19	LIU Y, KUAI X Y, YUAN X J, et al. Learning-based iterative interference cancellation for cognitive Internet of Things. IEEE Internet of Things Journal, 2019, 6(4): 7213- 7224. doi: 10.1109/JIOT.2019.2915598
20	LIU X, JIA M, DING H. Uplink resource allocation for multicarrier grouping cognitive Internet of Things based on k-means learning. Ad Hoc Networks, 2020, 96, 102002. doi: 10.1016/j.adhoc.2019.102002
21	ZHAO K, XU H W, HUANG L Y, et al. Research on wireless communication distance test for mobile IoT devices[C]//Proceedings of the IEEE MTT-S International Microwave Workshop Series on Advanced Materials and Processes for RF and THz Applications (IMWS-AMP). Washington D. C., USA: IEEE Press, 2022: 1-3.
22	宋佰霖, 许华, 齐子森, 等. 一种基于深度强化学习的协同通信干扰决策算法. 电子学报, 2022, 50(6): 1301- 1309.
	SONG B L, XU H, QI Z S, et al. A collaborative communication jamming decision algorithm based on deep reinforcement learning. Acta Electronica Sinica, 2022, 50(6): 1301- 1309.
23	GENDERS W, RAZAVI S. Evaluating reinforcement learning state representations for adaptive traffic signal control. Procedia Computer Science, 2018, 130, 26- 33. doi: 10.1016/j.procs.2018.04.008
24	WANG H J, GAO W, WANG W, et al. Research on obstacle avoidance planning for UUV based on A3C algorithm. Journal of Marine Science and Engineering, 2023, 12(63): 1- 14.
25	POKHREL S R. Learning from data streams for automation and orchestration of 6G industrial IoT: toward a semantic communication framework. Neural Computing and Applications, 2022, 34(18): 15197- 15206. doi: 10.1007/s00521-022-07065-z
26	ALI SHAH H, ZHAO L, KIM I M. Joint network control and resource allocation for space-terrestrial integrated network through hierarchal deep actor-critic reinforcement learning. IEEE Transactions on Vehicular Technology, 2021, 70(5): 4943- 4954. doi: 10.1109/TVT.2021.3071983
27	罗志强, 王伟, 朱晓荣. 基于A3C的无线异构网络自适应视频流传输控制方法. 电信科学, 2020, 36(12): 65- 76.
	LUO Z Q, WANG W, ZHU X R. An adaptive video stream transmission control method for wireless heterogeneous networks based on A3C. Telecommunications Science, 2020, 36(12): 65- 76.
28	HE Y, WANG Y H, QIU C, et al. Blockchain-based edge computing resource allocation in IoT: a deep reinforcement learning approach. IEEE Internet of Things Journal, 2021, 8(4): 2226- 2237. doi: 10.1109/JIOT.2020.3035437
29	邹玮琦, 牛朝阳, 刘伟, 等. 基于A3C的多功能雷达认知干扰决策方法. 系统工程与电子技术, 2023, 45(1): 86- 92.
	ZOU W Q, NIU C Y, LIU W, et al. Cognitive jamming decision-making method against multifunctional radar based on A3C. Systems Engineering and Electronics, 2023, 45(1): 86- 92.
30	WEI Q L, WANG L X, LIU Y, et al. Optimal elevator group control via deep asynchronous actor-critic learning. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(12): 5245- 5256. doi: 10.1109/TNNLS.2020.2965208
31	GUO S A, ZHAO X H. Deep reinforcement learning optimal transmission algorithm for cognitive Internet of Things with RF energy harvesting. IEEE Transactions on Cognitive Communications and Networking, 2022, 8(2): 1216- 1227. doi: 10.1109/TCCN.2022.3142727

[1]	LI Qinan, XUE Zhihao, ZHANG Xuejun. Improved Fast-HotStuff Blockchian Consensus Algorithm [J]. Computer Engineering, 2021, 47(8): 14-21.
[2]	ZHOU Shuo, QIU Runhe. Power Allocation Algorithm of CR-NOMA System Under Imperfect SIC [J]. Computer Engineering, 2021, 47(8): 195-200,209.
[3]	LU Yi, JIA Xiangdong, JI Pengshan, Lü Yaping. Research on Throughput of UAV-Assisted Multi-Tier Millimeter-Wave Heterogeneous Networks [J]. Computer Engineering, 2021, 47(7): 176-182.
[4]	GENG Junjie, LI Xiaoming, YAN Jinyao. Optimization of DASH System Based on Network Traffic Prediction [J]. Computer Engineering, 2021, 47(5): 292-300.
[5]	LU Yi, FU Jielin, QIU Hongbing, LIN Jiming, LI Yanlong. Idle Time Slot Reservation TDMA Protocol for Flying Ad-Hoc Networks [J]. Computer Engineering, 2021, 47(3): 202-208.
[6]	FENG Jianxin, LUAN Shuaishuai, LIU Junmei, PAN Chengsheng. An Unmanned Aerial Vehicle Trajectory Planning Method with High Throughput [J]. Computer Engineering, 2021, 47(1): 172-181.
[7]	WANG Shuai, YANG Hengxin, YANG Hua. Tree Anti-Collision Algorithm Based on Pseudo ID Code [J]. Computer Engineering, 2020, 46(4): 177-182.
[8]	LI Xinying, HAO Hao, HUANG Haiyan. Research on Cognitive Relay Network Performance Based on Simultaneous Information and Power Transfer [J]. Computer Engineering, 2020, 46(11): 201-206,213.
[9]	YANG Guanzhi, CHEN Pengfei, CUI Xinkai, HOU Weiyan. Overview and Performance Test of NB-IoT [J]. Computer Engineering, 2020, 46(1): 1-14.
[10]	XU Changbiao, GUO Ruibo, XIAN Yongju. Deployment Scheme of In-Band Backhaul Base Station in Heterogeneous Network [J]. Computer Engineering, 2020, 46(1): 150-156,163.
[11]	SUN Zhenyu, SHI Jingyan, SUN Gongxing, DU Ran, JIANG Xiaowei, ZOU Jiaheng, TAN Hongnan. Dual Layer Job Scheduling System for Large Scale Heterogeneous Computing Clusters [J]. Computer Engineering, 2020, 46(1): 187-195.
[12]	XU Bin, HE Yucheng. High Throughput Layered Decoding Design of QC-LDPC Codes [J]. Computer Engineering, 2019, 45(7): 121-125,133.
[13]	TIAN Xinji,ZHANG Haotian. User-centeric semi-dynamic clustering method for JT-CoMP scenarios [J]. Computer Engineering, 2019, 45(6): 134-139,145.
[14]	LIU Lihua,GUAN Wu,LIANG Liping. Design of Parallel High Throughput Multi-mode Polar Codes Encoder [J]. Computer Engineering, 2019, 45(4): 72-77.
[15]	SUN Yanzan,FAN Weirong,ZHANG Shunqing,WANG Tao,WU Yating. Resource Allocation Algorithm for Dense D2D Network Based on Graph Coloring [J]. Computer Engineering, 2019, 45(2): 26-31.

Please choose a citation manager

Content to export