用于热成像数据的卷积神经网络特征图筛选方法

doi:10.19678/j.issn.1000-3428.0069176

摘要/Abstract

摘要：

红外热成像数据可以有效辅助可见光图像数据, 弥补其在天气和光照条件上的不足。现有的研究往往借助域适应将基于可见光图像数据训练得到的卷积神经网络用于处理热成像数据, 以弥补热成像数据缺少大量标注训练集的不足, 但是这类方法仍无法避免一定程度的训练。而一些研究者发现, 图像在频域上呈现域不变成分和随域改变成分的分离现象。受这一现象的启发, 提出一种基于离散余弦变换和卡方独立性分数的卷积神经网络特征图筛选方法。利用频域分离域不变成分和随域改变成分, 借鉴卡方独立性检验的思想提出基于频段分量的独立性分数, 用于度量特征图的差异度, 使用聚类将特征图分类, 保留主要包含域不变成分的特征图分支, 得到适用于热成像数据的网络。实验结果表明, 该方法可以充分利用预训练卷积神经网络的潜在预测能力, 且不需要重新训练模型。预训练网络无法预测热成像数据, 而筛选后的网络前5位预测结果与目标相关的比例最高可达90%。

关键词: 热成像数据, 离散余弦变换, 域适应, 卷积神经网络, 交通场景

Abstract:

Commonly used visible light image data may fail because of harsh weather or poor light conditions; however, infrared thermal imaging data can effectively complement visible light image data. Existing studies often rely on Domain Adaptation(DA) to apply Convolutional Neural Network(CNN) to visible light image data to process thermal imaging data and overcome the lack of a large annotated training set for infrared data. However, DA methods cannot completely avoid the training process. Researchers have found that the domain-invariant and domain-variant components of an image can be separated in the frequency domain. Inspired by this phenomenon, a filtering method for feature maps from CNN based on Discrete Cosine Transform(DCT) and chi-square independence index is proposed. The domain-invariant and domain-variant components are separated in the frequency domain. By imitating the chi-square independence test, an independence index based on frequency components is proposed to measure the degree of difference in feature maps. According to this index, clustering is used to classify feature maps and identify the class(es) to be maintained or dropped. Thereafter, neural network suitable for thermal imaging data is constructed. The experimental results indicate that this method can determine the latent capabilities of pre-trained CNN for visible light images to extract the features of thermal imaging data without retraining the network. Although the pre-trained network failed to predict the thermal imaging data, the network constructed using the proposed method achieves up to 90% matching between the object and the top five prediction results.

Key words: thermal imaging data, Discrete Cosine Transform(DCT), Domain Adaptation(DA), Convolutional Neural Network(CNN), traffic scene

张雷, 沈国琛, 欧冬秀. 用于热成像数据的卷积神经网络特征图筛选方法[J]. 计算机工程, 2024, 50(4): 31-40.

Lei ZHANG, Guochen SHEN, Dongxiu OU. Filtering Method for Feature Maps from Convolutional Neural Network for Thermal Imaging Data[J]. Computer Engineering, 2024, 50(4): 31-40.

https://www.ecice06.com/CN/Y2024/V50/I4/31

图/表 15

图1 特征图筛选流程

Fig.1 Procedure of feature maps filtering

图2 预训练网络浅层特征图

Fig.2 Low-level feature maps from pre-trained network

图3 特征图频域基频置零前后对比图

Fig.3 Comparison of the original frequency domain diagram of the feature map and the frequency domain diagram with the base frequency been set to zero

图4 典型特征图的频域图

Fig.4 Frequency domain diagrams of typical feature maps

图5 人工置零后的浅层特征图

Fig.5 Low-level feature maps after manual zeroing

图6 近邻传播算法聚类结果

Fig.6 Clustering result using affinity propagation algorithm

图7 DBSCAN算法聚类结果

Fig.7 Clustering result using DBSCAN algorithm

参考文献 25

1	刘泽. 自动驾驶汽车多源异构传感器环境感知方法研究[D]. 镇江: 江苏大学, 2022.
	LIU Z. Research on environment perception method of multi-source heterogeneous sensors for autonomous vehicle[D]. Zhenjiang: Jiangsu University, 2022. (in Chinese)
2	张珊. 基于多源数据融合的三维目标检测研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
	ZHANG S. Research on 3D target detection based on multi-source data fusion[D]. Harbin: Harbin Institute of Technology, 2020. (in Chinese)
3	TAN H J, OU D X, ZHANG L, et al. Infrared sensation-based salient targets enhancement methods in low-visibility scenes. Sensors, 2022, 22 (15): 5835. doi: 10.3390/s22155835
4	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/1409.1556.pdf.
5	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
6	DAYAL A, AISHWARYA M, ABHILASH S, et al. Adversarial unsupervised domain adaptation for hand gesture recognition using thermal images. IEEE Sensors Journal, 2023, 23 (4): 3493- 3504. doi: 10.1109/JSEN.2023.3235379
7	AKKAYA I B, ALTINEL F, HALICI U. Self-training guided adversarial domain adaptation for thermal imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 4317-4326.
8	KIM Y H, SHIN U, PARK J, et al. MS-UDA: multi-spectral unsupervised domain adaptation for thermal image semantic segmentation. IEEE Robotics and Automation Letters, 2021, 6 (4): 6497- 6504. doi: 10.1109/LRA.2021.3093652
9	USTUN B, KAYA A K, CAKIR A, et al. Spectral transfer guided active domain adaptation for thermal imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 449.
10	GAN L, LEE C, CHUNG S. Unsupervised RGB-to-thermal domain adaptation via multi-domain attention network[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2210.04367.pdf.
11	MARNISSI M A, FRADI H, SAHBANI A, et al. Unsupervised thermal-to-visible domain adaptation method for pedestrian detection. Pattern Recognition Letters, 2022, 153 (C): 222- 231.
12	MUNIR F, AZAM S, JEON M. SSTN: self-supervised domain adaptation thermal object detection for autonomous driving[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2103.03150v2.pdf.
13	VS V, POSTER D, YOU S Y, et al. Meta-UDA: unsupervised domain adaptive thermal object detection using meta-learning[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2022: 3697.
14	KIEU M, BAGDANOV A D, BERTINI M. Bottom-up and layerwise domain adaptation for pedestrian detection in thermal images[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 17(1): 32.
15	KIEU M, BAGDANOV A D, BERTINI M, et al. Domain adaptation for privacy-preserving pedestrian detection in thermal imagery[C]//Proceedings of International Conference on Image Analysis and Processing. Berlin, Germany: Springer, 2019: 203-213.
16	GRAZIANI M, ANDREARCZYK V, MÜLLER H. Visualizing and interpreting feature reuse of pretrained CNNs for histopathology[C]//Proceedings of IMVIP 2019: Irish Machine Vision and Image Processing Conference Proceedings. Dublin, Ireland: Technological University Dublin, 2019: 28.
17	HERRMANN C, RUF M, BEYERER J, et al. CNN-based thermal infrared person detection by domain adaptation[M]//MICHAEL C D. Autonomous systems: sensors, vehicles, security, and the internet of everything. Orlando, USA: SPIE, 2018: 38-43.
18	GUO T T, HUYNH C P, SOLH M. Domain-adaptive pedestrian detection in thermal images[C]//Proceedings of the IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2019: 1660-1664.
19	VERTENS J, ZÜRN J, BURGARD W. HeatNet: bridging the day-night domain gap in semantic segmentation with thermal images[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2003.04645.pdf.
20	MUNIR F, AZAM S, RAFIQUE M, et al. Thermal object detection using domain adaptation through style consistency[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2006.00821v2.pdf.
21	PRAJAPATI K, CHUDASAMA V, PATEL H, et al. Channel split convolutional neural network(ChaSNet) for thermal image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 4363-4372.
22	DU J M, LU H Z, HU M F, et al. CNN-based infrared dim small target detection algorithm using target-oriented shallow-deep features and effective small anchor. IET Image Processing, 2021, 15 (1): 1- 15. doi: 10.1049/ipr2.12001
23	HUANG J X, GUAN D Y, XIAO A R, et al. FSDR: frequency space domain randomization for domain generalization[EB/OL]. [2023-13-17]. https://arxiv.org/pdf/2103.02370.pdf.
24	XU M J, QIN L Y, CHEN W J, et al. Multi-view adversarial discriminator: mine the non-causal factors for object detection in unseen domains[EB/OL]. [2023-13-17]. https://arxiv.org/pdf/2304.02950.pdf.
25	章永来, 周耀鉴. 聚类算法综述. 计算机应用, 2019, 39 (7): 1869- 1882. URL
	ZHANG Y L, ZHOU Y J. Review of clustering algorithms. Journal of Computer Applications, 2019, 39 (7): 1869- 1882. URL

[1]	王志浩, 钱沄涛. 基于Swin Transformer的双流遥感图像时空融合超分辨率重建[J]. 计算机工程, 2024, 50(9): 33-45.
[2]	李俊俊, 董建刚, 李坤. 基于Kubernetes的集群节能策略研究[J]. 计算机工程, 2024, 50(9): 82-91.
[3]	张鲁, 田春伟, 宋焕生, 刘侍刚. 用于低剂量CT图像去噪的多级双树复小波网络[J]. 计算机工程, 2024, 50(9): 266-275.
[4]	高煜宝, 文志诚. 基于注意力机制的双路解码器图像去噪方法[J]. 计算机工程, 2024, 50(9): 324-332.
[5]	王蕾, 党时鹏, 潘丰. 基于卷积神经网络的隐匿性旁路预测模型[J]. 计算机工程, 2024, 50(8): 40-49.
[6]	耿丽丽, 牛保宁. 基于通道相似度熵的卷积神经网络裁剪[J]. 计算机工程, 2024, 50(7): 133-143.
[7]	张洋, 刘畅, 李少青. 基于可控制性度量的图神经网络门级硬件木马检测方法[J]. 计算机工程, 2024, 50(7): 164-173.
[8]	牛瑞婷, 严天峰, 高锐, 王映植. 低信噪比下基于深度学习TCNN-MobileNet的调制识别[J]. 计算机工程, 2024, 50(7): 204-215.
[9]	张溢文, 蔡满春, 陈咏豪, 朱懿, 姚利峰. 融合空间特征的多尺度深度伪造检测方法[J]. 计算机工程, 2024, 50(7): 240-250.
[10]	逯焕宇, 张永宏, 马光义, 谢东林, 田伟. 基于半监督对抗学习的遥感图像水体提取[J]. 计算机工程, 2024, 50(7): 251-263.
[11]	魏琢艺, 罗迈, 李文兵, 曾远松, 余伟江, 杨跃东. 基于多源域适应的单细胞智能分类[J]. 计算机工程, 2024, 50(6): 48-55.
[12]	于洋, 孙芳芳, 吕华, 李扬, 王晓民. 基于多尺度时空注意力网络的微表情检测方法[J]. 计算机工程, 2024, 50(6): 228-235.
[13]	张雷, 沈国琛, 欧冬秀. 用于热成像数据的卷积神经网络特征图筛选方法[J]. 计算机工程, 2024, 50(4): 31-40.
[14]	李政学, 李枝名, 彭德中, 陈杰. 基于特征对比学习和图卷积的社交网络用户分类[J]. 计算机工程, 2024, 50(4): 258-266.
[15]	姜百浩, 刘静, 仇大伟, 姜良. 深度学习在脊柱图像分割中的应用综述[J]. 计算机工程, 2024, 50(3): 1-15.

选择文件类型/文献管理软件名称

选择包含的内容