Optimized FANET Routing Algorithm with Reinforcement Learning Based on Function Approximation

doi:10.19678/j.issn.1000-3428.0059591

Abstract

Abstract: The high-speed movement of nodes in Flying Ad-Hoc Network(FANET) has caused difficulties in maintaining the links of the FANET routing protocol.To address the problem,an algorithm named QLA-OLSR is proposed based on Reinforcement Learning(RL) for adaptive optimization of link state routing.By sensing the changing number of the node neighbors and the service loads in the dynamic environment,the Q-learning algorithm in RL is used to construct a value function.On this basis,the optimal HELLO time slot is solved to improve the performance of the node in link detection and maintenance.Then the State Similarity Mechanism(SSM) of the improved Kanerva coding algorithm is used to reduce the complexity of the algorithm while increasing its stability. Simulation results show that the QLA-OLSR algorithm can significantly improve the network throughput,reduce the overhead of routine maintenance,and is capable of self-learning.It is suitable for FANET in a highly dynamic environment.

Key words: Flying Ad-Hoc Network(FANET), function approximation, Q-learning, routing algorithm, adaptive HELLO time slot

摘要： 针对高速移动状态下的飞行自组网路由协议链路维护困难问题，提出一种基于强化学习的自适应链路状态路由优化算法QLA-OLSR。借鉴强化学习中的Q学习算法，通过感知动态环境下节点邻居数量变化和业务负载程度，构建价值函数求解最优HELLO时隙，提高节点链路发现与维护能力。利用优化Kanerva编码算法的状态相似度机制，降低QLA-OLSR算法复杂度并增强稳定性。仿真结果表明，QLA-OLSR算法能有效提升网络吞吐量，减少路由维护开销，且具有自学习特性，适用于高动态环境下的飞行自组网。

关键词: 飞行自组网, 函数逼近, Q学习, 路由算法, 自适应HELLO时隙

CLC Number:

TN929.52

XIE Yongsheng, YANG Yuwang, QIU Xiulin, WANG Yinyin. Optimized FANET Routing Algorithm with Reinforcement Learning Based on Function Approximation[J]. Computer Engineering, 2021, 47(11): 207-213.

谢勇盛, 杨余旺, 邱修林, 王吟吟. 基于函数逼近的强化学习FANET路由优化算法[J]. 计算机工程, 2021, 47(11): 207-213.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0059591

http://www.ecice06.com/EN/Y2021/V47/I11/207

Figures/Tables 8

References

[1] 陈林星,曾曦,曹毅.移动Ad Hoc网络:自组织分组无线网络技术[M].北京:电子工业出版社,2006. CHEN L X,ZENG X,CAO Y.Mobile Ad Hoc network:self-organizing packet wireless network technology[M].Beijing:Publishing House of Electronics Industry,2006.(in Chinese)
[2] GUPTA L,JAIN R,VASZKUN G.Survey of important issues in UAV communication networks[J].IEEE Communications Surveys & Tutorials,2016,18(2):1123-1152.
[3] GUILLEN-PEREZ A,CANO M D.Flying ad hoc networks:a new domain for network communications[J].Sensors,2018,18(10):3571.
[4] 孟利民,宋文波.移动自组网路由协议研究[M].北京:人民邮电出版社,2012. MENG L M,SONG W B.Mobile ad hoc network protocols[M].Beijing:Posts & Telecom Press,2012:21-28.(in Chinese)
[5] BEKMEZCI I,SAHINGOZ O K,TEMEL S.Flying Ad-Hoc Networks(FANETs):a survey[J].Ad Hoc Networks,2013,11(3):1254-1270.
[6] GOMEZ C,CATALAN M,MANTECON X,et al.Evaluating performance of real ad-hoc networks using AODV with Hello message mechanism for maintaining local connectivity[C]//Proceedings of the 16th International Symposium on Personal,Indoor and Mobile Radio Communications.Washington D.C.,USA:IEEE Press,2005:1327-1331.
[7] MAHMUD I,CHO Y Z.Adaptive Hello interval in FANET routing protocols for green UAVs[J].IEEE Access,2019,7:63004-63015.
[8] CLAUSEN T,DEARLOVE C,DEAN J.Mobile Ad Hoc Network(MANET) Neighborhood Discovery Protocol(NHDP):RFC 6130[R].Fremont,USA:IETF,2011.
[9] HERNANDEZ-CONS N,KASAHARA S,TAKAHASHI Y.Dynamic Hello/Timeout timer adjustment in routing protocols for reducing overhead in MANETs[J].Computer Communications,2010,33(15):1864-1878.
[10] GIRUKA V C,SINGHAL M.Hello protocols for ad-hoc networks:overhead and accuracy tradeoffs[C]//Proceedings of the 6th IEEE International Symposium on a World of Wireless Mobile and Multimedia Networks.Washington D.C.,USA:IEEE Press,2005:354-361.
[11] HAN S Y,LEE D.An adaptive Hello messaging scheme for neighbor discovery in on-demand MANET routing protocols[J].IEEE Communications Letters,2013,17(5):1040-1043.
[12] WATKINS C J C H,DAYAN P.Technical note:Q-learning[J].Machine Learning,1992,8(3):279-292.
[13] SUTTON R S,BARTO A G.Reinforcement learning:an introduction[M].Cambridge,USA:MIT Press,1998.
[14] 李世宝,肖雪松,刘建航,等.基于道路分段的车载自组织网络路由协议[J].计算机工程,2019,45(2):32-37. LI S B,XIAO X S,LIU J H,et al.Vehicular ad hoc network routing protocol based on road-subsection[J].Computer Engineering,2019,45(2):32-37.(in Chinese)
[15] LI W,ZHOU F,CHOWDHURY K R,et al.QTCP:adaptive congestion control with reinforcement learning[J].IEEE Transactions on Network Science and Engineering,2019,6(3):445-458.
[16] WU C,WANG Y M.Learning from big data:a survey and evaluation of approximation technologies for large-scale reinforcement learning[C]//Proceedings of 2017 IEEE International Conference on Computer and Information Technology.Washington D.C.,USA:IEEE Press,2017:1-8.
[17] LI W,MELEIS W.Adaptive adjacency Kanerva coding for memory-constrained reinforcement learning[C]//Proceedings of International Conference on Machine Learning and Data Mining in Pattern Recognition.Berlin,Geramany:Springer,2018:1-16.
[18] ASCHENBRUCK N,ERNST R,GERHARDS-PADILLA E,et al.BonnMotion:a mobility scenario generation and analysis tool[C]//Proceedings of the 3rd International ICST Conference on Simulation Tools and Techniques.New York,USA:ACM Press,2010:1-10.
[19] ROY R R.Handbook of mobile ad hoc networks for mobility models[M].Berlin,Germany:Springer,2011.
[20] CAMP T,BOLENG J,DAVIES V.A survey of mobility models for ad hoc network research[J].Wireless Communications and Mobile Computing,2002,2(5):483-502.
[21] 洪洁.高动态飞行器自组织网络关键技术研究[D].北京:中国科学院大学,2019. HONG J.Research on key technologies of highly dynamic flying ad hoc networks[D].Beijing:University of Chinese Academy of Sciences,2019.

Please choose a citation manager

Content to export