基于提示信息传输框架的图文行人重识别

doi:10.19678/j.issn.1000-3428.0252150

摘要/Abstract

摘要： 在基于图文的行人重识别任务中,基于图文预训练模型的参数初始化已成为主流范式,这有效突破了单模态模型因跨模态信息缺失导致的特征对齐瓶颈。现有方法聚焦于挖掘图像-文本联合嵌入空间中不同尺度下的语义特征进行优化,但新对齐范式的引入易使原模型在微调过程中陷入局部最优。为了解决上述问题,提出一种基于提示的信息传输(PIT)框架,通过在单模态编码器和跨模态图像-文本编码器的原始前向过程中嵌入跨模态提示标识符,促进早期特征融合,隐式地引导模型更加聚焦于模态不变的信息。PIT框架包含基于跨模态提示的对比学习(CL)损失和提示训练策略(PTS)。基于跨模态提示的CL损失旨在通过约束图文特征之间的相似度,构建兼具模态内区分度与模态间语义一致性的共享特征嵌入空间。PTS可以视为一种自蒸馏方法,通过将无提示特征与基准真相产生的伪目标视为另一种行人图文对的特征视图,监督跨模态提示特征的训练过程,使最终学习到的特征嵌入相较于无提示特征包含更丰富的多模态信息。实验结果表明,PIT在完全微调的基础上仅需要添加0.61×10⁶的参数,在CUHK-PEDES、ICFG-PEDES和RSTPReid数据集上所提模型相比基线模型的Rank-1提高了1.48、1.5 和1.55百分点。

关键词: 跨模态, 图文的行人重识别, 提示学习, BLIP模型, 自蒸馏学习

Abstract: In Image—Text person Re-IDentification (ITReID) tasks, initializing models with parameters from pretraining models has become a mainstream paradigm that effectively alleviates the feature alignment bottleneck of single-modal models caused by a lack of cross-modal information. Existing methods focus on mining semantic features at different scales in an image-text joint embedding space for optimization. However, the introduction of the new alignment paradigm is prone to causing the pretraining model to fall to a local minimum during fine-tuning. To address these issues, this paper proposes a Prompt-based Information Transfer (PIT) framework. By introducing cross-modal prompt tokens into the original forward process of the single-modal encoder and cross-modal image-text encoder, the proposed framework promotes early feature fusion and implicitly guides the pretraining model to focus more on modal-invariant information. PIT includes a prompt-based Contrastive Learning (CL) loss and Prompt Training Strategy (PTS). Prompt-based contrastive loss aims to construct a shared feature-embedding space with both intra-modal discrimination and inter-modal semantic consistency by constraining the similarity between graphic and text features. The PTS can be regarded as a form of self-distillation that treats the pseudo-targets generated by nonprompt features and ground truth as another view of image-text pairs, supervises the training process, and makes the learned embeddings contain richer multimodal information. Only 0.61×10⁶ additional parameters are introduced based on fine-tuning, and PIT achieves Rank-1 improvements of 1.48, 1.5, and 1.55 percentage points on three public datasets.

Key words: cross-modality, Image-Text person Re-IDentification (ITReID), prompt learning, BLIP model, self-distillation learining

中图分类号:

TP391

耿霞, 林贤文, 杨治. 基于提示信息传输框架的图文行人重识别[J]. 计算机工程, 2026, 52(10): 242-251.

GENG Xia, LIN Xianwen, YANG Zhi. Prompt-Based Information Transfer Framework for Image-Text Person Re-Identification[J]. Computer Engineering, 2026, 52(10): 242-251.

https://www.ecice06.com/CN/Y2026/V52/I10/242

参考文献

[1] LI S, XIAO T, LI H S, et al. Person search with natural language description[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Washington D.C., USA: IEEE Press, 2017: 1970-1979.
[2] HAN X Y, ZHONG X, HUANG W X, et al. See what you seek: semantic contextual integration for cloth-changing person re-identification[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2412.01345.
[3] RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of International Conference on Machine Learning. New York, USA: [s.n], 2021: 8748-8763.
[4] LI J N, SELVARAJU R R, GOTMARE A D, et al. Align before fuse: vision and language representation learning with momentum distillation[EB/OL].[2025-01-15]. https://arxiv.org/pdf/2107.07651.
[5] LI J N, LI D X, XIONG C M, et al. BLIP: bootstrapping language-image pre-training for unified vision-language understanding and generation[EB/OL]. [2025-01-15]. https://arxiv.org/pdf/2201.12086.
[6] JIANG D, YE M. Cross-modal implicit relation reasoning and aligning for text-to-image person retrieval[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 2787-2797.
[7] YAN S L, DONG N, ZHANG L Y, et al. CLIP-driven fine-grained text-image person re-identification[J]. IEEE Transactions on Image Processing, 2023, 32: 6032-6046.
[8] LIU Y T, LI Y W, LIU Z M, et al. CLIP-based synergistic knowledge transfer for text-based person retrieval[C]//Proceedings of IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2024: 7935-7939.
[9] SUN J T, FEI H, ZHENG Z D, et al. From data deluge to data curation: a filtering-WoRA paradigm for efficient text-based person search[EB/OL]. [2025-01-15]. https://arxiv.org/pdf/2404.10292.
[10] YANG S Y, ZHOU Y N, ZHENG Z D, et al. Towards unified text-based person retrieval: a large-scale multi-attribute and language search benchmark[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 4492-4501.
[11] HE K M, FAN H Q, WU Y X, et al. Momentum contrast for unsupervised visual representation learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2020: 9726-9735.
[12] 孟琪翔, 高志霖,王劲滔,等. 面向公共场所敏感目标与人体异常行为协同识别网络[J]. 光电工程, 2025, 52(8): 250131. MENG Q X, GAO Z L, WANG J T, et al. Collaborative detection network for sensitive targets and abnormal human behaviour in public places[J]. Opto-Electronic Engineering, 2025, 52(8): 250131. (in Chinese)
[13] DING Z F, DING C X, SHAO Z Y, et al. Semantically self-aligned network for text-to-image part-aware person re-identification[EB/OL]. [2025-01-15]. https://arxiv.org/pdf/2107.12666.
[14] ZHU A C, WANG Z J, LI Y F, et al. DSSL: deep surroundings-person separation learning for text-based person retrieval[C]//Proceedings of the 29th ACM International Conference on Multimedia. New York, USA: ACM Press, 2021: 209-217.
[15] ZHENG Z D, ZHENG L, GARRETT M, et al. Dual-path convolutional image-text embeddings with instance loss[J]. ACM Transactions on Multimedia Computing, Communications, and Applications, 2020, 16(2): 1-23.
[16] 姜定, 叶茫. 面向跨模态文本到图像行人重识别的Transformer网络[J]. 中国图象图形学报, 2023, 28(5): 1384-1395. JIANG D, YE M. Transformer network for cross-modal text-to-image person re-identification[J]. Journal of Image and Graphics, 2023, 28(5): 1384-1395. (in Chinese)
[17] YAN S L, DONG N, LIU J, et al. Learning comprehensive representations with richer self for text-to-image person re-identification[C]//Proceedings of the 31st ACM International Conference on Multimedia. New York, USA: ACM Press, 2023: 6202-6211.
[18] WANG Z J, ZHU A C, XUE J Y, et al. CAIBC: capturing all-round information beyond color for text-based person retrieval[C]//Proceedings of the 30th ACM International Conference on Multimedia. New York, USA: ACM Press, 2022: 5314-5322.
[19] 王晋溪, 鲁鸣鸣. 基于场景图知识的文本到图像行人重识别[J]. 模式识别与人工智能, 2024, 37(11): 947-959. WANG J X, LU M M. Scene graph knowledge based text-to-image person re-identification[J]. Pattern Recognition and Artificial Intelligence, 2024, 37(11): 947-959. (in Chinese)
[20] BAI Y, CAO M, GAO D M, et al. RaSa: relation and sensitivity aware representation learning for text-based person search[C]//Proceedings of the 32nd International Joint Conference on Artificial Intelligence. New York, USA: ACM Press, 2023: 555-563.
[21] KHATTAK M U, RASHEED H, MAAZ M, et al. MaPLe: multi-modal prompt learning[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2023: 19113-19122.
[22] LIN D X, PENG Y X, MENG J K, et al. Cross-modal adaptive dual association for text-to-image person retrieval[J]. IEEE Transactions on Multimedia, 2024, 26: 6609-6620.
[23] YE M, SHEN J B, LIN G J, et al. Deep learning for person re-identification: a survey and outlook[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022, 44(6): 2872-2893.
[24] CHEN Y H, ZHANG G Q, LU Y J, et al. TIPCB: a simple but effective part-based convolutional baseline for text-based person search[J]. Neurocomputing, 2022, 494: 171-181.
[25] FUJII T, TARASHIMA S. BiLMa: bidirectional local-matching for text-based person re-identification[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops (ICCVW). Washington D.C., USA: IEEE Press, 2023: 2778-2782.
[26] QIN Y, CHEN Y K, PENG D Z, et al. Noisy-correspondence learning for text-to-image person re-identification[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2024: 27187-27196.
[27] CAO M, BAI Y, ZENG Z Y, et al. An empirical study of CLIP for text-based person search[C]// Proceedings of the AAAI Conference on Artificial Intelligence. [S.l.]: AAAI Press, 2024: 465-473.
[28] ERGASTI A, FONTANINI T, FERRARI C, et al. MARS: paying more attention to visual attributes for text-based person search[J]. ACM Transactions on Multimedia Computing Communications and Applications, 2025, 21(10):1-22.
[29] DENG Y C, HU Z P, HAN J K, et al. DualFocus: integrating plausible descriptions in text-based person re-identification[EB/OL]. [2025-01-15]. https://arxiv.org/pdf/2405.07459.

选择文件类型/文献管理软件名称

选择包含的内容