Filtering Method for Feature Maps from Convolutional Neural Network for Thermal Imaging Data

doi:10.19678/j.issn.1000-3428.0069176

Abstract

Abstract:

Commonly used visible light image data may fail because of harsh weather or poor light conditions; however, infrared thermal imaging data can effectively complement visible light image data. Existing studies often rely on Domain Adaptation(DA) to apply Convolutional Neural Network(CNN) to visible light image data to process thermal imaging data and overcome the lack of a large annotated training set for infrared data. However, DA methods cannot completely avoid the training process. Researchers have found that the domain-invariant and domain-variant components of an image can be separated in the frequency domain. Inspired by this phenomenon, a filtering method for feature maps from CNN based on Discrete Cosine Transform(DCT) and chi-square independence index is proposed. The domain-invariant and domain-variant components are separated in the frequency domain. By imitating the chi-square independence test, an independence index based on frequency components is proposed to measure the degree of difference in feature maps. According to this index, clustering is used to classify feature maps and identify the class(es) to be maintained or dropped. Thereafter, neural network suitable for thermal imaging data is constructed. The experimental results indicate that this method can determine the latent capabilities of pre-trained CNN for visible light images to extract the features of thermal imaging data without retraining the network. Although the pre-trained network failed to predict the thermal imaging data, the network constructed using the proposed method achieves up to 90% matching between the object and the top five prediction results.

Key words: thermal imaging data, Discrete Cosine Transform(DCT), Domain Adaptation(DA), Convolutional Neural Network(CNN), traffic scene

摘要：

红外热成像数据可以有效辅助可见光图像数据, 弥补其在天气和光照条件上的不足。现有的研究往往借助域适应将基于可见光图像数据训练得到的卷积神经网络用于处理热成像数据, 以弥补热成像数据缺少大量标注训练集的不足, 但是这类方法仍无法避免一定程度的训练。而一些研究者发现, 图像在频域上呈现域不变成分和随域改变成分的分离现象。受这一现象的启发, 提出一种基于离散余弦变换和卡方独立性分数的卷积神经网络特征图筛选方法。利用频域分离域不变成分和随域改变成分, 借鉴卡方独立性检验的思想提出基于频段分量的独立性分数, 用于度量特征图的差异度, 使用聚类将特征图分类, 保留主要包含域不变成分的特征图分支, 得到适用于热成像数据的网络。实验结果表明, 该方法可以充分利用预训练卷积神经网络的潜在预测能力, 且不需要重新训练模型。预训练网络无法预测热成像数据, 而筛选后的网络前5位预测结果与目标相关的比例最高可达90%。

关键词: 热成像数据, 离散余弦变换, 域适应, 卷积神经网络, 交通场景

Lei ZHANG, Guochen SHEN, Dongxiu OU. Filtering Method for Feature Maps from Convolutional Neural Network for Thermal Imaging Data[J]. Computer Engineering, 2024, 50(4): 31-40.

张雷, 沈国琛, 欧冬秀. 用于热成像数据的卷积神经网络特征图筛选方法[J]. 计算机工程, 2024, 50(4): 31-40.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069176

http://www.ecice06.com/EN/Y2024/V50/I4/31

Figures/Tables 15

Fig.1 Procedure of feature maps filtering

Fig.2 Low-level feature maps from pre-trained network

Fig.3 Comparison of the original frequency domain diagram of the feature map and the frequency domain diagram with the base frequency been set to zero

Fig.4 Frequency domain diagrams of typical feature maps

Fig.5 Low-level feature maps after manual zeroing

Fig.6 Clustering result using affinity propagation algorithm

Fig.7 Clustering result using DBSCAN algorithm

References 25

1	刘泽. 自动驾驶汽车多源异构传感器环境感知方法研究[D]. 镇江: 江苏大学, 2022.
	LIU Z. Research on environment perception method of multi-source heterogeneous sensors for autonomous vehicle[D]. Zhenjiang: Jiangsu University, 2022. (in Chinese)
2	张珊. 基于多源数据融合的三维目标检测研究[D]. 哈尔滨: 哈尔滨工业大学, 2020.
	ZHANG S. Research on 3D target detection based on multi-source data fusion[D]. Harbin: Harbin Institute of Technology, 2020. (in Chinese)
3	TAN H J, OU D X, ZHANG L, et al. Infrared sensation-based salient targets enhancement methods in low-visibility scenes. Sensors, 2022, 22 (15): 5835. doi: 10.3390/s22155835
4	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/1409.1556.pdf.
5	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 770-778.
6	DAYAL A, AISHWARYA M, ABHILASH S, et al. Adversarial unsupervised domain adaptation for hand gesture recognition using thermal images. IEEE Sensors Journal, 2023, 23 (4): 3493- 3504. doi: 10.1109/JSEN.2023.3235379
7	AKKAYA I B, ALTINEL F, HALICI U. Self-training guided adversarial domain adaptation for thermal imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 4317-4326.
8	KIM Y H, SHIN U, PARK J, et al. MS-UDA: multi-spectral unsupervised domain adaptation for thermal image semantic segmentation. IEEE Robotics and Automation Letters, 2021, 6 (4): 6497- 6504. doi: 10.1109/LRA.2021.3093652
9	USTUN B, KAYA A K, CAKIR A, et al. Spectral transfer guided active domain adaptation for thermal imagery[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2023: 449.
10	GAN L, LEE C, CHUNG S. Unsupervised RGB-to-thermal domain adaptation via multi-domain attention network[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2210.04367.pdf.
11	MARNISSI M A, FRADI H, SAHBANI A, et al. Unsupervised thermal-to-visible domain adaptation method for pedestrian detection. Pattern Recognition Letters, 2022, 153 (C): 222- 231.
12	MUNIR F, AZAM S, JEON M. SSTN: self-supervised domain adaptation thermal object detection for autonomous driving[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2103.03150v2.pdf.
13	VS V, POSTER D, YOU S Y, et al. Meta-UDA: unsupervised domain adaptive thermal object detection using meta-learning[C]//Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2022: 3697.
14	KIEU M, BAGDANOV A D, BERTINI M. Bottom-up and layerwise domain adaptation for pedestrian detection in thermal images[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 17(1): 32.
15	KIEU M, BAGDANOV A D, BERTINI M, et al. Domain adaptation for privacy-preserving pedestrian detection in thermal imagery[C]//Proceedings of International Conference on Image Analysis and Processing. Berlin, Germany: Springer, 2019: 203-213.
16	GRAZIANI M, ANDREARCZYK V, MÜLLER H. Visualizing and interpreting feature reuse of pretrained CNNs for histopathology[C]//Proceedings of IMVIP 2019: Irish Machine Vision and Image Processing Conference Proceedings. Dublin, Ireland: Technological University Dublin, 2019: 28.
17	HERRMANN C, RUF M, BEYERER J, et al. CNN-based thermal infrared person detection by domain adaptation[M]//MICHAEL C D. Autonomous systems: sensors, vehicles, security, and the internet of everything. Orlando, USA: SPIE, 2018: 38-43.
18	GUO T T, HUYNH C P, SOLH M. Domain-adaptive pedestrian detection in thermal images[C]//Proceedings of the IEEE International Conference on Image Processing. Washington D. C., USA: IEEE Press, 2019: 1660-1664.
19	VERTENS J, ZÜRN J, BURGARD W. HeatNet: bridging the day-night domain gap in semantic segmentation with thermal images[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2003.04645.pdf.
20	MUNIR F, AZAM S, RAFIQUE M, et al. Thermal object detection using domain adaptation through style consistency[EB/OL]. [2023-12-17]. https://arxiv.org/pdf/2006.00821v2.pdf.
21	PRAJAPATI K, CHUDASAMA V, PATEL H, et al. Channel split convolutional neural network(ChaSNet) for thermal image super-resolution[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 4363-4372.
22	DU J M, LU H Z, HU M F, et al. CNN-based infrared dim small target detection algorithm using target-oriented shallow-deep features and effective small anchor. IET Image Processing, 2021, 15 (1): 1- 15. doi: 10.1049/ipr2.12001
23	HUANG J X, GUAN D Y, XIAO A R, et al. FSDR: frequency space domain randomization for domain generalization[EB/OL]. [2023-13-17]. https://arxiv.org/pdf/2103.02370.pdf.
24	XU M J, QIN L Y, CHEN W J, et al. Multi-view adversarial discriminator: mine the non-causal factors for object detection in unseen domains[EB/OL]. [2023-13-17]. https://arxiv.org/pdf/2304.02950.pdf.
25	章永来, 周耀鉴. 聚类算法综述. 计算机应用, 2019, 39 (7): 1869- 1882. URL
	ZHANG Y L, ZHOU Y J. Review of clustering algorithms. Journal of Computer Applications, 2019, 39 (7): 1869- 1882. URL

[1]	Xinlin XIE, Dongxu YIN, Taoyuan ZHANG, Gang XIE. Multiscale Fusion Crowd Counting Algorithm Based on Attention Mechanism [J]. Computer Engineering, 2024, 50(3): 290-297.
[2]	Baihao JIANG, Jing LIU, Dawei QIU, Liang JIANG. Review of Deep Learning Applications in Spinal Image Segmentation [J]. Computer Engineering, 2024, 50(3): 1-15.
[3]	Bohan WANG, Xiaoyan JIANG, Liuyi FAN. Semantic Segmentation Improvement Method Based on Deep Supervision for the Construction of Latent Space [J]. Computer Engineering, 2024, 50(3): 191-199.
[4]	Haochen XU, Manhua LIU. Facial Landmark Detection Based on Hierarchical Self-Attention Network [J]. Computer Engineering, 2024, 50(2): 239-246.
[5]	Qian LI, Weibo WEI, Guangyu YANG, Jintao SONG, Lu SUN, Zhenkuan PAN. Poisson Denoising Variational Model Based on Prior-Driven Deep Neural Network [J]. Computer Engineering, 2024, 50(2): 273-280.
[6]	Shangwang BAI, Mengyao WANG, Jing HU, Zhibo CHEN. Multi-Region Attention Network for Fine-Grained Image Classification [J]. Computer Engineering, 2024, 50(1): 271-278.
[7]	Xianguo LI, Bin LI. Image Deblurring Based on Transformer and Multi-scale CNN [J]. Computer Engineering, 2023, 49(9): 226-233, 245.
[8]	Yixiao DU, Hongjun WANG, Xiuhe LI. Research on Fingerprint Positioning Method of Radiation Source Based on Spectrum Map [J]. Computer Engineering, 2023, 49(9): 183-190, 198.
[9]	Lu HAN, Weigang HUO, Yonghui ZHANG, Tao LIU. Multivariate Time Series Forecasting Based on Multi-Scale Feature Fusion and Dual-Attention Mechanism [J]. Computer Engineering, 2023, 49(9): 99-108.
[10]	SHEN Xueli, TIAN Guiyuan, JIANG Yanji, MA Linlin. Time-Frequency Domain Speech Enhancement Algorithm Based on Dual-Stage Conv-Transformer [J]. Computer Engineering, 2023, 49(6): 123-130.
[11]	DING Zixuan, YU Lei, ZHANG Juan, LI Xiang, WANG Xinyu. Image Super-Resolution Reconstruction Based on Depth Residual Adaptive Attention Network [J]. Computer Engineering, 2023, 49(5): 231-238.
[12]	YANG Jingjing, XIE Haiyan, XUE Nini, ZHANG Aoming. Research on Underwater Image Denoising Based on Dual-Channels Residual Network [J]. Computer Engineering, 2023, 49(4): 188-198.
[13]	CHEN Zhixu, JIN Yanxia, LU Ye, YANG Jing, LIU Yabian, SHI Zhiru. Multi-Precision Clothing Modeling Method Based on Subgraph Convolutional Neural Network [J]. Computer Engineering, 2023, 49(4): 174-181.
[14]	ZHONG Baorong, WU Xialing. Research on Lightweight Human Pose Estimation Based on High-Resolution Network [J]. Computer Engineering, 2023, 49(4): 226-232,239.
[15]	LIU Jingjng, HUANG Hao. Fundamental Frequency Extraction Model Using Convolutional Neural Networks with Non-local Modules [J]. Computer Engineering, 2023, 49(3): 128-133,160.

Please choose a citation manager

Content to export