Cross-Modality Person Re-identification Using Four-Stream Network Based on Dual-Intermediate Modalities

doi:10.19678/j.issn.1000-3428.0065333

Abstract

Abstract:

Most cameras are equipped with infrared and visible light functions. Therefore, the application of re-identification methods will inevitably solve the problem of cross-modality person re-identification. To reduce the difference between infrared and visible light modes in cross-modality person re-identification and improve recognition accuracy, a four-stream cross-modality person re-identification method based on dual-intermediate modalities is proposed. Two lightweight networks generate dual-intermediate modalities images of visible light and infrared modes, respectively, inherit labels from visible light and infrared images, and reconstruct a network suitable for learning shared features of four modalities by splitting ResNet50 backbone network. Additionally, the problem of parameter sharing in four-stream networks is also explored, and the impact of the number of four modalities shared blocks on cross-modality person re-identification is analyzed. The experimental results show that when compared to HcTri, the proposed method increases Rank-1 and mAP by 2.38 and 4.64 percentage points, respectively, in global search mode on the SYSU-MM01 dataset, 6.24 and 6.77 percentage points, respectively, in indoor search mode.Compared to HcTri, the proposed method increases Rank-1, mAP and mINP by 2.52, 3.74, and 4.68 percentage points, respectively, in visible light to infrared search mode on the RegDB dataset, in the infrared to visible light search mode, Rank-1, mAP, and mINP increase by 2.70, 3.47, and 5.56 percentage points.

Key words: person re-identification, dual-intermediate modalities, four-stream backbone network, cross-modality re-identification, parameter sharing

摘要：

摄像头大多配备红外和可见光功能，因此，重识别方法的应用必然要解决跨模态行人重识别问题。为缩小跨模态行人重识别中红外和可见光模态之间的差异，提高识别精度，提出基于双中间模态的四流跨模态行人重识别方法。由2个轻量级网络分别生成可见光模态和红外模态的双中间模态图像，并从可见光图像和红外图像中继承标签，通过拆分ResNet50骨干网络以重构适应于4种模态共享特征学习的网络。此外，还探讨了四流骨干网络中的参数共享问题，分析四模态共享块数量对于跨模态行人重识别的影响。实验结果表明，相比HcTri，该方法在SYSU-MM01数据集上的全局检索模式下的Rank-1和mAP分别提高2.38和4.64个百分点，在室内检索模式下分别提高6.24和6.77个百分点，在RegDB数据集上可见光至红外检索模式下的Rank-1、mAP和mINP分别提高2.52、3.74和4.68个百分点，在红外至可见光检索模式下的Rank-1、mAP和mINP分别分别提高2.70、3.47和5.56个百分点。

关键词: 行人重识别, 双中间模态, 四流骨干网络, 跨模态重识别, 参数共享

Hua HAN, Li HUANG, Jin TIAN, Chunyuan WANG. Cross-Modality Person Re-identification Using Four-Stream Network Based on Dual-Intermediate Modalities[J]. Computer Engineering, 2023, 49(8): 302-309.

韩华, 黄丽, 田瑾, 王春媛. 基于双中间模态的四流网络跨模态行人重识别[J]. 计算机工程, 2023, 49(8): 302-309.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0065333

http://www.ecice06.com/EN/Y2023/V49/I8/302

Figures/Tables 9

References 29

1	董亚超, 刘宏哲, 徐成. 基于显著性多尺度特征协作融合的行人重识别方法. 计算机工程, 2021, 47(6): 234-244, 252 doi: 10.19678/j.issn.1000-3428.0057938
	DONG Y C, LIU H Z, XU C. Person re-identification method based on joint fusion of saliency multi-scale features. Computer Engineering, 2021, 47(6): 234-244, 252 doi: 10.19678/j.issn.1000-3428.0057938
2	HAN H A, MA W J, ZHOU M C, et al. A novel semi-supervised learning approach to pedestrian reidentification. IEEE Internet of Things Journal, 2021, 8(4): 3042- 3052. doi: 10.1109/JIOT.2020.3024287
3	HAN H A, ZHOU M C, SHANG X W, et al. KISS+ for rapid and accurate pedestrian re-identification. IEEE Transactions on Intelligent Transportation Systems, 2021, 22(1): 394- 403. doi: 10.1109/TITS.2019.2958741
4	罗浩, 姜伟, 范星, 等. 基于深度学习的行人重识别研究进展. 自动化学报, 2019, 45(11): 2032- 2049. doi: 10.16383/j.aas.c180154
	LUO H, JIANG W, FAN X, et al. A survey on deep learning based person re-identification. Acta Automatica Sinica, 2019, 45(11): 2032- 2049. doi: 10.16383/j.aas.c180154
5	陈丹, 李永忠, 于沛泽, 等. 跨模态行人重识别研究与展望. 计算机系统应用, 2020, 29(10): 20- 28. URL
	CHEN D, LI Y Z, YU P Z, et al. Research and prospect of cross modality person re-identification. Computer Systems & Applications, 2020, 29(10): 20- 28. URL
6	WU A C, ZHENG W S, YU H X, et al. RGB-infrared cross-modality person re-identification[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 5390-5399.
7	YE M, LAN X Y, LI J W, et al. Hierarchical discriminative learning for visible thermal person re-identification[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2018: 7501-7508.
8	ZHANG S Z, YANG Y F, WANG P, et al. Attend to the difference: cross-modality person re-identification via contrastive correlation. IEEE Transactions on Image Processing, 2021, 30, 8861- 8872. doi: 10.1109/TIP.2021.3120881
9	LIU H J, CHENG J, WANG W, et al. Enhancing the discriminative feature learning for visible-thermal cross-modality person re-identification. Neurocomputing, 2020, 398, 11- 19. doi: 10.1016/j.neucom.2020.01.089
10	LU Y, WU Y E, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 13379-13389.
11	HAO Y, WANG N N, LI J E, et al. HSME: hypersphere manifold embedding for visible thermal person re-identification[C]//Proceedings of the Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2019: 8385-8392.
12	YE M, WANG Z, LAN X Y, et al. Visible thermal person re-identification via dual-constrained top-ranking[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. New York, USA: ACM Press, 2018: 1092 -1099.
13	YE M, LAN X Y, WANG Z, et al. Bi-directional center-constrained top-ranking for visible thermal person re-identification. IEEE Transactions on Information Forensics and Security, 2020, 15, 407- 419. doi: 10.1109/TIFS.2019.2921454
14	李灏, 唐敏, 林建武, 等. 基于改进困难三元组损失的跨模态行人重识别框架. 计算机科学, 2020, 47(10): 180- 186. doi: 10.11896/jsjkx.191100061
	LI H, TANG M, LIN J W, et al. Cross-modality person re-identification framework based on improved hard triplet loss. Computer Science, 2020, 47(10): 180- 186. doi: 10.11896/jsjkx.191100061
15	DAI P Y, JI R R, WANG H B, et al. Cross-modality person re-identification with generative adversarial training[C]//Proceedings of the 27th International Joint Conference on Artificial Intelligence. New York, USA: ACM Press, 2018: 677-683.
16	WANG Z X, WANG Z, ZHENG Y Q, et al. Learning to reduce dual-level discrepancy for infrared-visible person re-identification[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 618-626.
17	WANG G A, ZHANG T Z, CHENG J, et al. RGB-infrared cross-modality person re-identification via joint pixel and feature alignment[C]//Proceedings of International Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 3622-3631.
18	CHOI S, LEE S M, KIM Y, et al. Hi-CMD: hierarchical cross-modality disentanglement for visible-infrared person re-identification[C]//Proceedings of Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 10254-10263.
19	LI D G, WEI X, HONG X P, et al. Infrared-visible cross-modal person re-identification with an X modality[C]//Proceedings of the AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2020: 4610-4617.
20	ZHU Y X, YANG Z, WANG L, et al. Hetero-center loss for cross-modality person re-identification. Neurocomputing, 2020, 386, 97- 109. doi: 10.1016/j.neucom.2019.12.100
21	LIU H J, TAN X H, ZHOU X C. Parameter sharing exploration and hetero center triplet loss for visible -thermal person re-identification [EB/OL]. [2022-06-14]: https://arxiv.org/pdf/2008.06223v1.pdf.
22	BASARAN E, GÖKMEN M, KAMASAK M E. An efficient framework for visible-infrared cross modality person re-identification. Signal Processing: Image Communication, 2020, 87, 115933. doi: 10.1016/j.image.2020.115933
23	ZHENG Z D, ZHENG L A, YANG Y. A discriminatively learned CNN embedding for person reidentification. ACM Transactions on Multimedia Computing, Communications, and Applications, 2018, 14(1): 1- 20.
24	NGUYEN D, HONG H, KIM K, et al. Person recognition system based on a combination of body images from visible light and thermal cameras. Sensors, 2017, 17(3): 605. doi: 10.3390/s17030605
25	LIU H J, CHAI Y X, TAN X H, et al. Strong but simple baseline with dual-granularity triplet loss for visible-thermal person re-identification. IEEE Signal Processing Letters, 2021, 28, 653- 657.
26	YE M, SHEN J B, LIN G J, et al. Deep learning for person re-identification: a survey and outlook [EB/OL]. [2022-06-14]: https://arxiv.org/abs/2001.04193v2.
27	YE M, SHEN J B, CRANDALL D J, et al. Dynamic dual-attentive aggregation learning for visible-infrared person re-identification[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 229-247.
28	WANG G A, ZHANG T Z, YANG Y, et al. Cross-modality paired-images generation for RGB-infrared person re-identification[C]//Proceedings of the 34th AAAI Conference on Artificial Intelligence. New Orleans, USA: AAAI Press, 2020: 12144-12151.
29	YE M, SHEN J B, SHAO L. Visible-infrared person re-identification via homogeneous augmented tri-modal learning. IEEE Transactions on Information Forensics and Security, 2021, 16, 728- 739.

方法	全局检索模式			室内检索模式
方法	Rank-1	mAP	mINP	Rank-1	mAP	mINP
Zero-Pad	14.80	15.95	—	20.58	26.92	—
Tone	12.52	14.42	—	20.82	26.38	—
cmGAN	26.97	27.80	—	31.63	42.19	—
HSME	20.68	23.12	—	—	—	—
D2RL	28.90	29.20	—	—	—	—
AlignGAN	42.40	40.70	—	45.90	54.30	—
DSCSN	35.10	37.40	—	—	—	—
HC^[20]	56.96	54.95	—	59.74	64.91	—
BDTR	27.32	27.32	—	31.92	41.86	—
eBDTR^[13]	27.82	28.42	—	32.46	42.46	—
EDFL	36.94	40.77	—	—	—	—
X modality	49.92	50.73	—	—	—	—
JSIA	38.10	36.90	—	43.80	52.90	—
DDAG	54.75	53.02	—	61.02	67.98	—
cm-SSFT	61.60	63.20	—	70.50	72.60	—
Hi-CMD	34.94	35.94	—	—	—	—
HAT	55.29	53.89	—	62.10	69.37	—
AGW	47.50	47.65	35.30	54.17	62.97	59.23
HcTri	61.58	56.91	41.11	62.65	67.35	62.41
DGTL	57.34	55.13	—	63.11	69.20	—
本文方法	63.96	61.55	43.95	68.89	74.12	72.06

方法	全局检索模式			室内检索模式
方法	Rank-1	mAP	mINP	Rank-1	mAP	mINP
Zero-Pad	14.80	15.95	—	20.58	26.92	—
Tone	12.52	14.42	—	20.82	26.38	—
cmGAN	26.97	27.80	—	31.63	42.19	—
HSME	20.68	23.12	—	—	—	—
D2RL	28.90	29.20	—	—	—	—
AlignGAN	42.40	40.70	—	45.90	54.30	—
DSCSN	35.10	37.40	—	—	—	—
HC^[20]	56.96	54.95	—	59.74	64.91	—
BDTR	27.32	27.32	—	31.92	41.86	—
eBDTR^[13]	27.82	28.42	—	32.46	42.46	—
EDFL	36.94	40.77	—	—	—	—
X modality	49.92	50.73	—	—	—	—
JSIA	38.10	36.90	—	43.80	52.90	—
DDAG	54.75	53.02	—	61.02	67.98	—
cm-SSFT	61.60	63.20	—	70.50	72.60	—
Hi-CMD	34.94	35.94	—	—	—	—
HAT	55.29	53.89	—	62.10	69.37	—
AGW	47.50	47.65	35.30	54.17	62.97	59.23
HcTri	61.58	56.91	41.11	62.65	67.35	62.41
DGTL	57.34	55.13	—	63.11	69.20	—
本文方法	63.96	61.55	43.95	68.89	74.12	72.06

方法	可见光至红外检索模式			红外至可见光检索模式
方法	Rank-1	mAP	mINP	Rank-1	mAP	mINP
Zero-Pad	17.75	18.90	—	16.63	17.82	—
HSME	50.85	47.00	—	50.15	46.16	—
D2RL	43.40	44.10	—	—	—	—
AlignGAN	57.90	53.60	—	56.30	53.40	—
DSCSN	60.80	60.00	—	—	—	—
BDTR	33.56	32.76	—	32.92	31.96	—
eBDTR^[13]	34.62	33.46	—	34.21	32.49	—
EDFL	52.58	52.98	—	51.89	52.13	—
X modality	62.21	60.18	—	—	—	—
JSIA	48.50	49.30	—	48.10	48.90	—
DDAG	69.34	63.46	—	68.06	61.80	—
cm-SSFT	72.30	72.90	—	71.00	71.70	—
Hi-CMD	70.93	66.04	—	—	—	—
HAT	71.83	67.56	—	70.02	66.30	—
AGW	70.05	66.37	50.19	—	—	—
HcTri	89.93	81.70	67.28	88.08	80.25	64.21
DGTL	83.92	73.78	—	81.59	71.65	—
本文方法	92.45	85.44	71.96	90.78	83.72	69.77

方法	可见光至红外检索模式			红外至可见光检索模式
方法	Rank-1	mAP	mINP	Rank-1	mAP	mINP
Zero-Pad	17.75	18.90	—	16.63	17.82	—
HSME	50.85	47.00	—	50.15	46.16	—
D2RL	43.40	44.10	—	—	—	—
AlignGAN	57.90	53.60	—	56.30	53.40	—
DSCSN	60.80	60.00	—	—	—	—
BDTR	33.56	32.76	—	32.92	31.96	—
eBDTR^[13]	34.62	33.46	—	34.21	32.49	—
EDFL	52.58	52.98	—	51.89	52.13	—
X modality	62.21	60.18	—	—	—	—
JSIA	48.50	49.30	—	48.10	48.90	—
DDAG	69.34	63.46	—	68.06	61.80	—
cm-SSFT	72.30	72.90	—	71.00	71.70	—
Hi-CMD	70.93	66.04	—	—	—	—
HAT	71.83	67.56	—	70.02	66.30	—
AGW	70.05	66.37	50.19	—	—	—
HcTri	89.93	81.70	67.28	88.08	80.25	64.21
DGTL	83.92	73.78	—	81.59	71.65	—
本文方法	92.45	85.44	71.96	90.78	83.72	69.77

拆分方法	全局检索模式			室内检索模式
拆分方法	Rank-1	mAP	mINP	Rank-1	mAP	mINP
$ {s}_{44111} $	61.31	56.53	37.68	63.46	68.56	64.85
$ {s}_{44211} $	63.69	59.60	41.35	66.35	71.30	67.69
$ {s}_{44221} $	59.52	56.79	39.58	64.74	70.33	65.51
$ {s}_{42111} $	62.07	58.15	39.02	66.39	71.28	67.07
$ {s}_{42211} $	61.80	57.81	40.23	65.91	70.64	67.75

Please choose a citation manager

Content to export