基于A3C的认知物联网通信干扰消除算法

doi:10.19678/j.issn.1000-3428.0068319

摘要/Abstract

摘要：

针对频谱资源干扰管理的智能化需求, 提出一种基于异步优势行动者-评论家(A3C)的干扰消除算法, 旨在应对认知物联网(CIoT)通信系统中由频谱资源共享引起的干扰问题。通过智能体的学习和优化, 帮助次级用户(SU)在受到干扰影响时做出最优的决策, 从而改善通信质量和系统性能。在该算法中, 当SU遭受干扰影响通信质量时, 智能体通过学习和优化, 使SU能够根据当前的位置信息、发射功率、接收功率以及干扰程度选择最低干扰程度的行动, 并执行该行动后获得的奖励。智能体通过尝试不同减少干扰的行动, 并根据奖励的反馈调整策略, 达到最大化定义干扰程度指标和信号质量指标的奖励函数的目的, 从而最大程度地减少干扰对通信质量的影响。实验结果表明, 与传统k-means算法以及深度递归Q网络(DRQN)和深度Q网络(DQN)优化算法相比, 基于A3C的干扰消除算法具有更短的收敛时间、更高的执行效率以及更高的系统吞吐量, 较3种基准方法在吞吐量性能上至少提高7%, 能够有效地减少干扰对通信质量的不利影响。

关键词: 认知物联网, 干扰消除, 异步优势行动者-评论家算法, 干扰程度, 信号质量, 吞吐量

Abstract:

To address the intelligent needs of spectrum resource interference management, in this study, an Asynchronous Advantage Actor-Critic(A3C)-based intelligent interference elimination algorithm is proposed to tackle the interference problem caused by spectrum resource sharing in a Cognitive Internet of Things (CIoT) communication system. Through learning and optimization of the agent, the algorithm helps Secondary Users (SU) make optimal decisions when affected by interference, thereby improving communication quality and system performance. When the communication quality of the SU is affected by interference, the learning and optimization of the agent enables the SU to choose the action with the lowest interference degree according to the current position information, transmit power, and receive power and interference degree, whereby a reward is offered after performing the action. Intelligent agents attempt different actions to reduce interference and adjust their strategies based on the reward feedback, thereby maximizing the reward function by considering the interference level and signal quality indicators and minimizing the impact of interference on communication quality. The simulation results show that compared with the traditional k-means, Deep Recurrent Q-Network (DRQN), and Deep Q-Network (DQN) optimization algorithms, the A3C-based interference cancellation algorithm has a shorter convergence time, higher execution efficiency, and at least 7% higher throughput performance than the three benchmark methods. This demonstrates that the proposed algorithm can effectively reduce the adverse effects of interference on communication quality.

Key words: Cognitive Internet of Things(CIoT), interference cancellation, Asynchronous Advantage Actor-Critic(A3C) algorithm, interference level, signal quality, throughput

刘新梦, 谢健骊, 李翠然, 王亦鸣. 基于A3C的认知物联网通信干扰消除算法[J]. 计算机工程, 2024, 50(10): 281-290.

LIU Xinmeng, XIE Jianli, LI Cuiran, WANG Yiming. A3C-Based Interference Cancellation Algorithm for Cognitive Internet of Things Communication[J]. Computer Engineering, 2024, 50(10): 281-290.

https://www.ecice06.com/CN/Y2024/V50/I10/281

图/表 12

图1 CIoT通信系统模型示意图

Fig.1 Schematic diagram of CIoT communication system model

图2 干扰消除整体框架

Fig.2 Overall framework for interference elimination

图3 基站及用户分布图

Fig.3 Distribution map of base stations and users

图4 在不同学习率下本文算法的收敛情况

Fig.4 Convergence of algorithms in this paper under different learning rates

图5 不同算法的累积时间对比

Fig.5 Cumulative time comparison of different algorithms

图6 干扰功率与系统吞吐量关系

Fig.6 Relationship between interference power and system throughput

图7 系统吞吐量与SU数的关系

Fig.7 The relationship between system throughput and the number of SUs

图8 系统吞吐量与SU-PU距离之间的关系

Fig.8 Relationship between system throughput and SU-PU distance

图9 系统吞吐量与SU-SU距离之间的关系

Fig.9 Relationship between system throughput and SU-SU distance

参考文献 31

1	LIU X, DING H, HU S. Uplink resource allocation for NOMA-based hybrid spectrum access in 6G-enabled cognitive Internet of Things. IEEE Internet of Things Journal, 2021, 8(20): 15049- 15058. doi: 10.1109/JIOT.2020.3007017
2	ZAHEER K, OTHMAN M, REHMANI M H, et al. A survey of decision-theoretic models for Cognitive Internet of Things (CIoT). IEEE Access, 2018, 6, 22489- 22512. doi: 10.1109/ACCESS.2018.2825282
3	WANG D Y, QI P H, FU Q F, et al. Multiple high-order cumulants-based spectrum sensing in full-duplex-enabled cognitive IoT networks. IEEE Internet of Things Journal, 2021, 8(11): 9330- 9343. doi: 10.1109/JIOT.2021.3055782
4	EJAZ W, IBNKAHLA M. Multiband spectrum sensing and resource allocation for IoT in cognitive 5G networks. IEEE Internet of Things Journal, 2018, 5(1): 150- 163. doi: 10.1109/JIOT.2017.2775959
5	LI F, LAM K Y, LI X H, et al. Advances and emerging challenges in cognitive Internet-of-Things. IEEE Transactions on Industrial Informatics, 2020, 16(8): 5489- 5496. doi: 10.1109/TII.2019.2953246
6	BALA I, AHUJA K. Energy-efficient framework for throughput enhancement of cognitive radio network by exploiting transmission mode diversity. Journal of Ambient Intelligence and Humanized Computing, 2023, 14(3): 2167- 2184. doi: 10.1007/s12652-021-03428-x
7	HE T J, CHIN K W, SOH S, et al. A novel distributed resource allocation scheme for wireless-powered cognitive radio Internet of Things networks. IEEE Internet of Things Journal, 2021, 8(20): 15486- 15499. doi: 10.1109/JIOT.2021.3071396
8	DENG J H, CHEN S H, KU M L. Multiuser MIMO precoders with proactive primary interference cancelation and link quality enhancement for cognitive radio relay systems. IEEE Access, 2017, 5, 17701- 17712. doi: 10.1109/ACCESS.2017.2749122
9	张梓扬, 常军, 黄一帆, 等. 基于强化学习的空间引力波探测望远镜系统外杂光抑制研究. 光电工程, 2024, 51(2): 71- 81.
	ZHANG Z Y, CHAGN J, HUAGN Y F, et al. Reinforcement learning-based stray light suppression study for space-based gravitational wave detection telescope system. Opto-Electronic Engineering, 2024, 51(2): 71- 81.
10	SUN C, DING H, LIU X. Multichannel spectrum access based on reinforcement learning in cognitive Internet of Things. Ad Hoc Networks, 2020, 106, 102200. doi: 10.1016/j.adhoc.2020.102200
11	TAN X, ZHOU L, WANG H J, et al. Cooperative multi-agent reinforcement-learning-based distributed dynamic spectrum access in cognitive radio networks. IEEE Internet of Things Journal, 2022, 9(19): 19477- 19488. doi: 10.1109/JIOT.2022.3168296
12	MOAYEDIAN N S, SALEHI S, KHABBAZIAN M. Fair resource allocation in cooperative cognitive radio IoT networks. IEEE Access, 2020, 8, 191067- 191079. doi: 10.1109/ACCESS.2020.3032204
13	SAFDAR MALIK T, RAZZAQ MALIK K, AFZAL A, et al. RL-IoT: reinforcement learning-based routing approach for cognitive radio-enabled IoT communications. IEEE Internet of Things Journal, 2023, 10(2): 1836- 1847. doi: 10.1109/JIOT.2022.3210703
14	XU L, YIN W X, ZHANG X L, et al. Fairness-aware throughput maximization over cognitive heterogeneous NOMA networks for industrial cognitive IoT. IEEE Transactions on Communications, 2020, 68(8): 4723- 4733. doi: 10.1109/TCOMM.2020.2992720
15	TULI S, ILAGER S, RAMAMOHANARAO K, et al. Dynamic scheduling for stochastic edge-cloud computing environments using A3C learning and residual recurrent neural networks. IEEE Transactions on Mobile Computing, 2022, 21(3): 940- 954. doi: 10.1109/TMC.2020.3017079
16	王毅然, 经小川, 贾福凯, 等. 基于多智能体协同强化学习的多目标追踪方法. 计算机工程, 2020, 46(11): 90- 96. doi: 10.3778/j.issn.1002-8331.1911-0132
	WANG Y R, JING X C, JIA F K, et al. Multi-target tracking method based on multi-agent collaborative reinforcement learning. Computer Engineering, 2020, 46(11): 90- 96. doi: 10.3778/j.issn.1002-8331.1911-0132
17	杨思明, 单征, 丁煜, 等. 深度强化学习研究综述. 计算机工程, 2021, 47(12): 19- 29. URL
	YANG S M, SHAN Z, DING Y, et al. Survey of research on deep reinforcement learning. Computer Engineering, 2021, 47(12): 19- 29. URL
18	LIU Y, YUAN X J, LIANG Y C, et al. Machine learning based iterative detection and multi-interference cancellation for cognitive IoT. IEEE Communications Letters, 2020, 24(9): 1995- 1999. doi: 10.1109/LCOMM.2020.2997048
19	LIU Y, KUAI X Y, YUAN X J, et al. Learning-based iterative interference cancellation for cognitive Internet of Things. IEEE Internet of Things Journal, 2019, 6(4): 7213- 7224. doi: 10.1109/JIOT.2019.2915598
20	LIU X, JIA M, DING H. Uplink resource allocation for multicarrier grouping cognitive Internet of Things based on k-means learning. Ad Hoc Networks, 2020, 96, 102002. doi: 10.1016/j.adhoc.2019.102002
21	ZHAO K, XU H W, HUANG L Y, et al. Research on wireless communication distance test for mobile IoT devices[C]//Proceedings of the IEEE MTT-S International Microwave Workshop Series on Advanced Materials and Processes for RF and THz Applications (IMWS-AMP). Washington D. C., USA: IEEE Press, 2022: 1-3.
22	宋佰霖, 许华, 齐子森, 等. 一种基于深度强化学习的协同通信干扰决策算法. 电子学报, 2022, 50(6): 1301- 1309.
	SONG B L, XU H, QI Z S, et al. A collaborative communication jamming decision algorithm based on deep reinforcement learning. Acta Electronica Sinica, 2022, 50(6): 1301- 1309.
23	GENDERS W, RAZAVI S. Evaluating reinforcement learning state representations for adaptive traffic signal control. Procedia Computer Science, 2018, 130, 26- 33. doi: 10.1016/j.procs.2018.04.008
24	WANG H J, GAO W, WANG W, et al. Research on obstacle avoidance planning for UUV based on A3C algorithm. Journal of Marine Science and Engineering, 2023, 12(63): 1- 14.
25	POKHREL S R. Learning from data streams for automation and orchestration of 6G industrial IoT: toward a semantic communication framework. Neural Computing and Applications, 2022, 34(18): 15197- 15206. doi: 10.1007/s00521-022-07065-z
26	ALI SHAH H, ZHAO L, KIM I M. Joint network control and resource allocation for space-terrestrial integrated network through hierarchal deep actor-critic reinforcement learning. IEEE Transactions on Vehicular Technology, 2021, 70(5): 4943- 4954. doi: 10.1109/TVT.2021.3071983
27	罗志强, 王伟, 朱晓荣. 基于A3C的无线异构网络自适应视频流传输控制方法. 电信科学, 2020, 36(12): 65- 76.
	LUO Z Q, WANG W, ZHU X R. An adaptive video stream transmission control method for wireless heterogeneous networks based on A3C. Telecommunications Science, 2020, 36(12): 65- 76.
28	HE Y, WANG Y H, QIU C, et al. Blockchain-based edge computing resource allocation in IoT: a deep reinforcement learning approach. IEEE Internet of Things Journal, 2021, 8(4): 2226- 2237. doi: 10.1109/JIOT.2020.3035437
29	邹玮琦, 牛朝阳, 刘伟, 等. 基于A3C的多功能雷达认知干扰决策方法. 系统工程与电子技术, 2023, 45(1): 86- 92.
	ZOU W Q, NIU C Y, LIU W, et al. Cognitive jamming decision-making method against multifunctional radar based on A3C. Systems Engineering and Electronics, 2023, 45(1): 86- 92.
30	WEI Q L, WANG L X, LIU Y, et al. Optimal elevator group control via deep asynchronous actor-critic learning. IEEE Transactions on Neural Networks and Learning Systems, 2020, 31(12): 5245- 5256. doi: 10.1109/TNNLS.2020.2965208
31	GUO S A, ZHAO X H. Deep reinforcement learning optimal transmission algorithm for cognitive Internet of Things with RF energy harvesting. IEEE Transactions on Cognitive Communications and Networking, 2022, 8(2): 1216- 1227. doi: 10.1109/TCCN.2022.3142727

[1]	李启南, 薛志浩, 张学军. 改进Fast-HotStuff区块链共识算法[J]. 计算机工程, 2021, 47(8): 14-21.
[2]	周烁, 仇润鹤. 不完全SIC下CR-NOMA系统的功率分配算法[J]. 计算机工程, 2021, 47(8): 195-200,209.
[3]	路艺, 贾向东, 纪澎善, 吕亚平. 多层无人机毫米波异构网络的吞吐量研究[J]. 计算机工程, 2021, 47(7): 176-182.
[4]	耿俊杰, 李晓明, 颜金尧. 基于网络流量预测的DASH系统优化[J]. 计算机工程, 2021, 47(5): 292-300.
[5]	陆毅, 符杰林, 仇洪冰, 林基明, 李燕龙. 适用于飞行自组网的闲置时隙预约TDMA协议[J]. 计算机工程, 2021, 47(3): 202-208.
[6]	冯建新, 栾帅帅, 刘俊梅, 潘成胜. 一种高吞吐量的无人机轨迹规划方法[J]. 计算机工程, 2021, 47(1): 172-181.
[7]	李新颖, 郝浩, 黄海燕. 基于信能同传的认知中继网络性能研究[J]. 计算机工程, 2020, 46(11): 201-206,213.
[8]	杨观止, 陈鹏飞, 崔新凯, 侯维岩. NB-IoT综述及性能测试[J]. 计算机工程, 2020, 46(1): 1-14.
[9]	徐昌彪, 郭瑞博, 鲜永菊. 异构网络下带内回程基站部署方案[J]. 计算机工程, 2020, 46(1): 150-156,163.
[10]	徐斌, 贺玉成. 高吞吐量QC-LDPC码分层译码设计[J]. 计算机工程, 2019, 45(7): 121-125,133.
[11]	田心记,张浩天. JT-CoMP场景中以用户为中心的半动态分簇方法[J]. 计算机工程, 2019, 45(6): 134-139,145.
[12]	孙彦赞,范卫蓉,张舜卿,王涛,吴雅婷. 基于图着色的密集D2D网络资源分配算法[J]. 计算机工程, 2019, 45(2): 26-31.
[13]	龙恳, 王鹏宇. 基于比例速率约束的NOMA公平资源分配算法[J]. 计算机工程, 2019, 45(12): 38-44.
[14]	刘建航, 李大伟, 黄庭培, 姜忠泰, 李世宝. 联合动态聚合帧长机制的速率自适应算法[J]. 计算机工程, 2019, 45(12): 91-97.
[15]	徐昌彪,吴杰. 超密集网中基于分簇的功率优化控制方案[J]. 计算机工程, 2019, 45(1): 55-60.

选择文件类型/文献管理软件名称

选择包含的内容