知识蒸馏Transformer的人物交互检测

doi:10.19678/j.issn.1000-3428.0069691

摘要/Abstract

摘要：

得到广泛应用的跨界之星Transformer, 在人-物交互(HOI)检测领域同样取得了很好的效果。基于此, 提出全新的基于知识蒸馏的Transformer(KDT)网络来进行端到端的HOI检测。由于Transformer网络建模的HOI整体特征粗糙, 针对HOI检测的3个子任务: 预测人框, 预测物框与物体类别, 预测人物之间的交互动作, 构建基础多分支Transformer结构, 包含一个人体实例分支、一个物体实例分支和一个交互分支, 并利用人、物分支的解码器为交互分支解码器提供人、物的区域线索。为了给Transformer结构提供关键的语义、空间信息, 预先生成物体类别和交互动词语义特征, 以及人物框的空间特征为不同的Transformer分支提供语义、空间线索, 进一步提升解码器对于不同HOI任务的特征提取能力。并在此基础上构建另一个多分支Transformer结构作为教师网络, 教师网络的解码器以预生成特征为解码器查询, 输出更精确的HOI特征。在训练过程中让基础多分支网络模仿教师网络的输出, 构建额外的类相似度损失度量两个网络输出预测之间的类内、类间向量相似度, 从而达到提升基础网络解码器性能的目的。实验结果表明, 在人-物交互基准数据集HICO-DET所有类别、稀有类别和非稀有类别上的均值平均精度(mAP)分别为32.13%、28.57%和33.19%, 对比基线取得了最多4.65百分点的提升。

关键词: Transformer网络, 人-物交互, 预生成特征, 教师网络, 类相似度损失

Abstract:

The widely used cross-field star Transformer has achieved good results in detecting Human-Object Interaction (HOI). This study proposes a new Transformer network, Knowledge Distillation-based Transformer (KDT), for HOI detection. Owing to the roughness of the overall HOI features modeled by the Transformer network, a basic multi-branched structure exists for the three tasks of HOI detection: prediction of human boxes, prediction of object boxes and object categories, and prediction of interaction categories. The basic multi-branched structure comprises a human instance branch, an object instance branch, and an interaction branch. Human and object branch decoders are used to provide interaction branch decoders with the regional tips of the human object. To provide key semantic and spatial information for the Transformer structure, the semantic features of the object categories and interaction verbs, as well as the spatial features of humans and object boxes, are generated to provide semantic and spatial tips for different Transformer branches, which further improves the feature extraction capability of the decoders. Next, the study proposes another multi-branched Transformer structure as a teacher network. The teacher network decoders output accurate HOI using the generated features as decoder queries. During the training process, the basic multi-branched network is allowed to imitate the output of the teacher network. Finally, the study presents an additional category similarity loss to measure the intra- and inter-category similarities between the output predictions of the two networks, thereby improving the performance of the basic network decoder. Experimental results show that the mean Average Precision (mAP) for all categories, rare categories, and non-rare categories on the HOI benchmark dataset HICO-DET are 32.13%, 28.57%, and 33.19%, respectively, achieving the highest increase of 4.65% compared with the baseline.

Key words: Transformer network, Human-Object Interaction (HOI), pre-generated feature, teacher network, category similarity loss

陈东吉, 赖惠成, 高古学, 马骏, 李俊凯, 权虎拓. 知识蒸馏Transformer的人物交互检测[J]. 计算机工程, 2026, 52(1): 206-216.

CHEN Dongji, LAI Huicheng, GAO Guxue, MA Jun, LI Junkai, QUAN Hutuo. Knowledge Distillation-based Transformer for Human-Object Interaction Detection[J]. Computer Engineering, 2026, 52(1): 206-216.

https://www.ecice06.com/CN/Y2026/V52/I1/206

图/表 14

图1 QPIC网络整体框架

Fig.1 The overall framework of QPIC network

图2 KDT网络整体框架

Fig.2 The overall framework of KDT network

图3 KDT训练策略

Fig.3 The training strategy of KDT

图4 解码器输出查询的类内、类间关系

Fig.4 The inter-category and intra-category relation of decoder output query

图5 基础网络各分支解码器交叉注意力的可视化

Fig.5 Visualization of the cross-attention of decoders in each branch of the basic network

图6 QPIC与KDT解码器交叉注意力的对比

Fig.6 Comparison of QPIC and KDT decoders cross-attention

图7 KDT的图像检测结果

Fig.7 The image detection results of KDT

图8 KDT的失败图像检测结果

Fig.8 The failed image detection results of KDT

参考文献 36

1	龚勋, 张志莹, 刘璐, 等. 人物交互检测研究进展综述. 西南交通大学学报, 2022, 57 (4): 693- 704.
	GONG X , ZHANG Z Y , LIU L , et al. A survey of human-object interaction detection. Journal of Southwest Jiaotong University, 2022, 57 (4): 693- 704.
2	REN S, HE K, GIRSHICK R, et al. Faster R-CNN: towards real-time object detection with region proposal networks[C]//Proceedings of the Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2015: 28-36.
3	GAO C, ZOU Y, HUANG J B. ICAN: Instance-centric attention network for human-object interaction detection[EB/OL]. [2024-03-01]. https://arxiv.org/abs/1808.10437.
4	ZHONG X B , DING C X , QU X , et al. Polysemy deciphering network for human-object interaction detection. Berlin, Germany: Springer, 2020.
5	ULUTAN O, IFTEKHAR A S M, MANJUNATH B S. VSGNet: spatial attention network for detecting human object interactions using graph convolutions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2020: 13617-13626.
6	LIAO Y, LIU S, WANG F, et al. PPDM: parallel point detection and matching for real-time human-object interaction detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE Press, 2020: 482-490.
7	KIM B , CHOI T , KANG J , et al. UnionDet: union-level detector towards real-time human-object interaction detection. Berlin, Germany: Springer, 2020.
8	CARION N, MASSA F, SYNNAEVE G, et al. End-to-end object detection with Transformers[C]//Proceedings of the European Conference on Computer Vision. Berlin, Germany: Springer, 2020: 213-229.
9	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2017: 130-241.
10	ZOU C, WANG B H, HU Y, et al. End-to-end human object interaction detection with HOI Transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE Press, 2021: 11825-11834.
11	TAMURA M, OHASHI H, YOSHINAGA T. QPIC: query-based pairwise human-object interaction detection with image-wide contextual information[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE Press, 2021: 10410-10419.
12	KIM B, LEE J, KANG J, et al. HOTR: end-to-end human-object interaction detection with Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2021: 74-83.
13	CHEN M F, LIAO Y, LIU S, et al. Reformulating HOI detection as adaptive set prediction[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE Press, 2021: 9004-9013.
14	DONG L Z, LI Z M, XU K L, et al. Category-aware transformer network for better human-object interaction detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE Press, 2022: 19538-19547.
15	IFTEKHAR A S M, CHEN H, KUNDU K, et al. What to look at and where: semantic and spatial refined Transformer for detecting human-object interactions[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE Press, 2022: 5353-5363.
16	白雪冰, 车进, 吴金蔓, 等. 基于Transformer视觉特征融合的图像描述方法. 计算机工程, 2024, 50 (8): 229- 238. doi: 10.19678/j.issn.1000-3428.0068402
	BAI X B , CHE J , WU J M , et al. Image captioning method based on Transformer visual features fusion. Computer Engineering, 2024, 50 (8): 229- 238. doi: 10.19678/j.issn.1000-3428.0068402
17	衡红军, 范昱辰, 王家亮. 基于Transformer的多方面特征编码图像描述生成算法. 计算机工程, 2023, 49 (2): 199- 205. doi: 10.19678/j.issn.1000-3428.0064450
	HENG H J , FAN Y C , WANG J L . Multifaceted feature coding image caption generation algorithm based on Transformer. Computer Engineering, 2023, 49 (2): 199- 205. doi: 10.19678/j.issn.1000-3428.0064450
18	RADFORD A, KIM J W, HALLACY C, et al. Learning transferable visual models from natural language supervision[C]//Proceedings of the International Conference on Machine Learning. Washington D. C., USA: IEEE Press, 2021: 8748-8763.
19	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. Las Vegas, USA: IEEE Press, 2016: 770-778.
20	REZATOFIGHI H, TSOI N, GWAK J, et al. Generalized intersection over union: a metric and a loss for bounding box regression[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE Press, 2019: 658-666.
21	LIN T Y, GOYAL P, GIRSHICK R, et al. Focal loss for dense object detection[C]//Proceedings of the IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 2980-2988.
22	CHAO Y W, LIU Y F, LIU X Y, et al. Learning to detect human-object interactions[C]//Proceedings of the IEEE Winter Conference on Applications of Computer Vision. Washington D. C., USA: IEEE Press, 2018: 381-389.
23	GUPTA S, MAILK J. Visual semantic role labeling[EB/OL]. [2024-03-01]. https://arxiv.org/abs/1505.04474.
24	LIN T Y, MAIRE M, BELONGIE S, et al. Microsoft COCO: common objects in context[C]//Proceedings of the 13th European Conference of Computer Vision. Berlin, Germany: Springer, 2014: 740-755.
25	LOSHCHILOV I, HUTTER F. Decoupled weight decay regularization[EB/OL]. [2024-03-01]. https://arxiv.org/abs/1711.05101.
26	XU K L , LI Z M , ZHANG Z J , et al. Effective actor-centric human-object interaction detection. Image and Vision Computing, 2022, 121, 104422. doi: 10.1016/j.imavis.2022.104422
27	LI Y L, ZHOU S Y, HUANG X J, et al. Transferable interactiveness knowledge for human-object interaction detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Long Beach, USA: IEEE Press, 2019: 3585-3594.
28	LI Y L, XU L, LIU X P, et al. PaStaNet: toward human activity knowledge engine[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Seattle, USA: IEEE Press, 2020: 382-391.
29	GAO C, XU J R, ZOU Y L, et al. DRG: dual relation graph for human-object interaction detection[C]// Proceedings of the 15th European Conference of Computer Vision. Berlin, Germany: Springer, 2020: 696-712.
30	LI Y L, LIU X, WU X, et al. Hoi analysis: integrating and decomposing human-object interaction[C]//Proceedings of the Advances in Neural Information Processing Systems. Cambridge, USA: MIT Press, 2020: 5011-5022.
31	HOU Z, PENG X J, QIAO Y, et al. Visual compositional learning for human-object interaction detection[C]//Proceedings of the 16th European Conference of Computer Vision. Berlin, Germany: Springer, 2020: 584-600.
32	ZHONG X B, QU X, DING C X, et al. Glance and gaze: inferring action-aware points for one-stage human-object interaction detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Nashville, USA: IEEE Press, 2021: 13234-13243.
33	DONG Q, TU Z W, LIAO H F, et al. Visual relationship detection using part-and-sum transformers with composite queries[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 3550-3559.
34	ZHANG F Z, CAMPBELL D, GOULD S. Spatially conditioned graphs for detecting human-object interactions[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2021: 13319-13327.
35	ZHANG F Z, CAMPBELL D, GOULD S. Efficient two-stage detection of human-object interactions with a novel unary-pairwise Transformer[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. New Orleans, USA: IEEE Press, 2022: 20104-20112.
36	KIM B, MUN J, ON K W, et al. MSTR: multi-scale Transformer for end-to-end human-object interaction detection[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2022: 19578-19587.

[1]	张天森, 徐晓娜, 赵悦, 张新宁. 基于级联Transformer和U-Net的MRI肝脏图像分割[J]. 计算机工程, 2025, 51(10): 308-318.
[2]	张池, 王忠, 姜添豪, 谢康民. 基于并行多注意力的语音增强网络[J]. 计算机工程, 2024, 50(4): 68-77.
[3]	王思成, 江浩, 陈晓. 具有跨尺度Transformer的高效多视图立体网络[J]. 计算机工程, 2024, 50(11): 266-275.
[4]	张炯, 王丽芳, 蔺素珍, 秦品乐, 米嘉, 刘阳. 局部全局特征耦合与交叉尺度注意的医学图像融合[J]. 计算机工程, 2023, 49(3): 238-247.
[5]	詹健浩, 甘利鹏, 毕永辉, 曾鹏, 李晓潮. 基于知识蒸馏的多模态融合行为识别方法[J]. 计算机工程, 2023, 49(10): 280-288, 297.
[6]	何涛, 俞舒曼, 徐鹤. 基于条件生成对抗网络与知识蒸馏的单幅图像去雾方法[J]. 计算机工程, 2022, 48(4): 165-172.

选择文件类型/文献管理软件名称

选择包含的内容