时域孪生网络融合Transformer的长时无人机视觉跟踪

doi:10.19678/j.issn.1000-3428.0068772

摘要/Abstract

摘要：

针对无人机(UAV)执行跟踪任务时经常出现尺寸变化、低分辨率、目标遮挡等场景导致跟踪目标框漂移的问题, 提出一种时域孪生网络融合Transformer的长时无人机视觉跟踪算法TTTrack。首先, 使用基于孪生网络的SiamFC++(AlexNet)算法作为基线算法; 其次, 利用Transformer自适应地提取历史帧的时空信息并在线更新模板, 从而将时空上下文信息储存为动态模板; 随后, 分别使用基准模板和动态模板与搜索特征图进行互相关运算, 获得响应图后利用Transformer融合两个响应图, 从而在连续帧之间建立时空上下文映射关系。实验结果表明, 在LaSOT长序列跟踪基准上TTTrack的成功率和精确率分别为63.9%和66.6%, 在UAV123跟踪基准上的成功率和精确率分别为61.4%和80.2%。与基线算法相比, 该算法在完全遮挡场景下的成功率和精确率分别提升7.4和8.0个百分点。TTTrack在DTB70跟踪基准上精确率达到82.1%, 并且跟踪速度为118帧/s, 满足实时性要求。测试结果验证了TTTrack具有良好的鲁棒性、实时性和抗干扰能力, 能有效应对长时UAV跟踪任务。

关键词: 时域孪生网络, Transformer模型, 无人机, 视觉跟踪, 时空信息

Abstract:

Frame drift often occurs when a Unmanned Aerial Vehicle (UAV) performs tracking tasks involving size changes, low resolution, and target occlusion. To that end, this study proposes a time-domain Siamese network fusion Transformer long-term UAV vision, which is called TTTrack. First, the SiamFC++ (AlexNet) algorithm based on the Siamese network is used as the baseline algorithm; Second, the Transformer is used to adaptively extract the spatio-temporal information of the historical frame and update the template online to store the spatio-temporal context information as a dynamic template; Third, the benchmark template is cross-correlated with the dynamic template and the search feature map is carried out to obtain two response maps; Finally, the Transformer is used to fuse the two response maps to establish a spatio-temporal context mapping relationship between consecutive frames. Based on the LaSOT long-sequence tracking benchmark, the success rate and accuracy of TTTrack are 63.9% and 66.6%, respectively. The success rate and accuracy of the UAV123 tracking benchmark are 61.4% and 80.2%, respectively. Compared with the baseline algorithm, the success rate and accuracy of this algorithm in fully occluded scenes increased by 7.4 percent and 8.0 percent points, respectively. TTTrack has an accuracy of 82.1% on the DTB70 tracking benchmark and a tracking speed of 118 frame/s, satisfying real-time requirements. The test results show that the proposed algorithm has good robustness, real-time performance, and anti-interference ability and can effectively handle long-term UAV tracking tasks.

Key words: time-domain Siamese network, Transformer model, Unmanned Aerial Vehicle (UAV), visual tracking, spatio-temporal information

谌海云, 余鹏, 王海川. 时域孪生网络融合Transformer的长时无人机视觉跟踪[J]. 计算机工程, 2024, 50(11): 107-118.

SHEN Haiyun, YU Peng, WANG Haichuan. Long-term UAV Vision Tracking with Time Domain Siamese Network Fusion Transformer[J]. Computer Engineering, 2024, 50(11): 107-118.

https://www.ecice06.com/CN/Y2024/V50/I11/107

图/表 13

图1 TTTrack跟踪器的整体框架

Fig.1 The overall framework of the TTTrack tracker

图2 自适应时空信息提取模块结构

Fig.2 Adaptive spatio-temporal information extraction module structure

图3 时空上下文映射模块结构

Fig.3 Spatio-temporal context mapping module structure

图4 不同算法在UAV123跟踪基准上的实验结果对比

Fig.4 Comparison of experimental results of different algorithms on the UAV123 tracking benchmark

图5 UAV123视频序列中不同算法的对比结果

Fig.5 Comparison results of different algorithms in UAV123 video sequence

图6 两种算法的热力图对比

Fig.6 Comparison of thermograms between two algorithms

图7 不同算法在LaSOT跟踪基准上的实验结果对比

Fig.7 Comparison of experimental results of different algorithms on the LaSOT tracking benchmark

图8 LaSOT视频序列中不同算法的对比结果

Fig.8 Comparison results of different algorithms in LaSOT video sequence

图9 不同算法在DTB70跟踪基准上的实验结果对比

Fig.9 Comparison of experimental results of different algorithms on DTB70 tracking benchmark

参考文献 31

1	闫超, 涂良辉, 王聿豪, 等. 无人机在我国民用领域应用综述. 飞行力学, 2022, 40(3): 1-6, 12.
	YAN C, TU L H, WANG L H, et al. Application of unmanned aerial vehicle in civil field in China. Flight Mechanics, 2022, 40(3): 1-6, 12.
2	孟琭, 杨旭. 目标跟踪算法综述. 自动化学报, 2019, 45(7): 1244- 1260.
	MENG L, YANG X. A survey of object tracking algorithms. Acta Automatica Sinica, 2019, 45(7): 1244- 1260.
3	YILMAZ A, JAVED O, SHAH M. Object tracking. ACM Computing Surveys, 2006, 38(4): 13. doi: 10.1145/1177352.1177355
4	钟莎, 黄玉清. 基于孪生区域候选网络的无人机指定目标跟踪. 计算机应用, 2021, 41(2): 523- 529.
	ZHONG S, HUANG Y Q. Specified object tracking of unmanned aerial vehicle based on Siamese region proposal network. Journal of Computer Applications, 2021, 41(2): 523- 529.
5	BERTINETTO L, VALMADRE J, HENRIQUES J F, et al. Fully-convolutional Siamese networks for object tracking[EB/OL]. [2023-10-05]. https://arxiv.org/abs/1606.09549.
6	LI B, YAN J J, WU W, et al. High performance visual tracking with Siamese region proposal network[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2018: 8971-8980.
7	ZHANG Z P, PENG H W. Deeper and wider Siamese networks for real-time visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 4591-4600.
8	XU Y D, WANG Z Y, LI Z X, et al. SiamFC++: towards robust and accurate visual tracking with target estimation guidelines. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12549- 12556. doi: 10.1609/aaai.v34i07.6944
9	CHEN Z D, ZHONG B N, LI G R, et al. Siamese box adaptive network for visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 6667-6676.
10	GUO D Y, WANG J, CUI Y, et al. SiamCAR: Siamese fully convolutional classification and regression for visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 6269-6277.
11	GUO D Y, SHAO Y Y, CUI Y, et al. Graph attention tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 9538-9547.
12	UASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2023-10-05]. https://arxiv.org/abs/1706.03762.
13	JACOB D, CHANG M, LEE K. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-10-05]. https://arxiv.org/abs/1810.04805.
14	WANG N, ZHOU W G, WANG J, et al. Transformer meets tracker: exploiting temporal context for robust visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 1571-1580.
15	CAO Z A, HUANG Z Y, PAN L, et al. TCTrack: temporal contexts for aerial tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 14798-14808.
16	YAN B, PENG H W, FU J L, et al. Learning spatio-temporal transformer for visual tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 10448-10457.
17	CUI Y T, JIANG C, WANG L M, et al. MixFormer: end-to-end tracking with iterative mixed attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 13608-13618.
18	SONG Z K, YU J Q, CHEN Y P P, et al. Transformer tracking with cyclic shifting window attention[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 8791-8800.
19	RUSSAKOVSKY O, DENG J, SU H, et al. ImageNet large scale visual recognition challenge. International Journal of Computer Vision, 2015, 115(3): 211- 252. doi: 10.1007/s11263-015-0816-y
20	HUANG L H, ZHAO X, HUANG K Q. GOT-10k: a large high-diversity benchmark for generic object tracking in the wild. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021, 43(5): 1562- 1577. doi: 10.1109/TPAMI.2019.2957464
21	FAN H, LIN L T, YANG F, et al. LaSOT: a high-quality benchmark for large-scale single object tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 5374-5383.
22	MUELLER M, SMITH N, GHANEM B. A benchmark and simulator for UAV tracking[EB/OL]. [2023-10-05]. https://link.springer.com/chapter/10.1007/978-3-319-46448-0_27.
23	LI S Y, YEUNG D Y. Visual object tracking for unmanned aerial vehicles: a benchmark and new motion models. Proceedings of the AAAI Conference on Artificial Intelligence, 2017, 31(1): 4140- 4146.
24	LI B, WU W, WANG Q, et al. SiamRPN++: evolution of Siamese visual tracking with very deep networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 4282-4291.
25	CAO Z A, FU C H, YE J J, et al. HiFT: hierarchical feature transformer for aerial tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 15457-15466.
26	DANELLJAN M, BHAT G, KHAN F S, et al. ECO: efficient convolution operators for tracking[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 6638-6646.
27	BHAT G, DANELLJAN M, VAN GOOL L, et al. Learning discriminative model prediction for tracking[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2019: 6182-6191.
28	FU Z H, LIU Q J, FU Z H, et al. STMTrack: template-free visual tracking with space-time memory networks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 13774-13783.
29	DANELLJAN M, VAN GOOL L, TIMOFTE R. Probabilistic regression for visual tracking[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 7183-7192.
30	DANELLJAN M, BHAT G, KHAN F S, et al. ATOM: accurate tracking by overlap maximization[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 4660-4669.
31	WANG Q, ZHANG L, BERTINETTO L, et al. Fast online object tracking and segmentation: a unifying approach[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2019: 1328-1338.

[1]	朱凯, 李理, 张彤, 江晟, 别一鸣. 基于Transformer的多阶段运动模糊图像修复网络[J]. 计算机工程, 2024, 50(9): 276-285.
[2]	王言国, 吕鹏远, 兰金江, 刘明哲, 秦冠军, 张硕桦, 周宇. 基于对抗训练与Transformer的风力发电机故障分类方法[J]. 计算机工程, 2024, 50(9): 377-384.
[3]	张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.
[4]	白雪冰, 车进, 吴金蔓, 陈玉敏. 基于Transformer视觉特征融合的图像描述方法[J]. 计算机工程, 2024, 50(8): 229-238.
[5]	赵宏, 王枭. 基于Swin-Transformer的黑色素瘤图像病灶分割研究[J]. 计算机工程, 2024, 50(8): 249-258.
[6]	秦媛, 张杭, 朱宏鹏, 李炯, 胡航. 卫星MIMO通信系统抗移动无人机集群干扰的在线BSS算法[J]. 计算机工程, 2024, 50(6): 65-76.
[7]	赵继达, 甄国涌, 储成群. 基于YOLOv8的无人机图像目标检测算法[J]. 计算机工程, 2024, 50(4): 113-120.
[8]	侯颖, 杨林, 胡鑫, 贺顺, 宋婉莹, 赵谦. 基于SwinT-YOLOX模型的自动扶梯行人安全检测算法[J]. 计算机工程, 2024, 50(3): 277-289.
[9]	吴嘉鑫, 孙一飞, 吴亚兰, 武继刚. 面向安全传输的低能耗无人机轨迹优化算法[J]. 计算机工程, 2024, 50(2): 59-67.
[10]	甘陈敏, 唐宏, 杨浩澜, 刘小洁, 刘杰. 融合卷积收缩门控的生成式文本摘要方法[J]. 计算机工程, 2024, 50(2): 98-104.
[11]	刘帅威, 李智, 王国美, 张丽. 基于Transformer和GAN的对抗样本生成算法[J]. 计算机工程, 2024, 50(2): 180-187.
[12]	赵健, 崔骞, 石佳, 刘岳. 基于文本和声学特征的双模态融合抑郁倾向识别算法[J]. 计算机工程, 2024, 50(11): 49-58.
[13]	张文政, 吴长悦, 满卫东, 刘明月, 赵文. 双通道对抗网络在建筑物分割中的应用[J]. 计算机工程, 2024, 50(11): 297-307.
[14]	万昊楠, 吴飞, 尹玲. 基于自适应无人机数量的节时部署优化算法[J]. 计算机工程, 2024, 50(10): 302-312.
[15]	池亚平, 岳梓岩, 林雨衡. 基于Transformer的SM4算法工作模式识别[J]. 计算机工程, 2023, 49(9): 109-117.

选择文件类型/文献管理软件名称

选择包含的内容