新闻类短视频关键帧摘要模型的研究与实现

doi:10.19678/j.issn.1000-3428.0065727

计算机工程 ›› 2023, Vol. 49 ›› Issue (8): 182-189. doi: 10.19678/j.issn.1000-3428.0065727

新闻类短视频关键帧摘要模型的研究与实现

崔晓丹¹, 刘达维¹, 刘逸凡¹, 赵志滨¹^,*, 任酉贵¹^,², 闫永明³

1. 东北大学计算机科学与工程学院, 沈阳 110169
2. 辽宁省自然资源事务服务中心, 沈阳 110001
3. 沈阳帝信人工智能产业研究院有限公司, 沈阳 110136

收稿日期:2022-09-13 出版日期:2023-08-15 发布日期:2022-12-09
通讯作者: 赵志滨
作者简介:
崔晓丹（1998－），女，硕士研究生，主研方向为计算机视觉、机器学习
刘达维，硕士研究生
刘逸凡，硕士研究生
任酉贵，博士研究生
闫永明，博士

Research and Implementation of Key Frame Summarization Model for News Short Video

Xiaodan CUI¹, Dawei LIU¹, Yifan LIU¹, Zhibin ZHAO¹^,*, Yougui REN¹^,², Yongming YAN³

1. School of Computer Science and Engineering, Northeastern University, Shenyang 110169, China
2. Service Center of Natural Resource Affairs of Liaoning Province, Shenyang 110001, China
3. Shenyang Dixin Artificial Intelligence Industry Research Institute Co., Ltd., Shenyang 110136, China

Received:2022-09-13 Online:2023-08-15 Published:2022-12-09
Contact: Zhibin ZHAO

摘要/Abstract

摘要：

根据传播学的“声画关系”理论，新闻类短视频通过音频直接有效地传达视频内容，属于典型的“主声说”视频。现有视频摘要技术忽略了声画关系对视频内容表现的影响，导致其在特定类型短视频摘要任务中效果不稳定。针对新闻类短视频“主声”的特点，提出基于多模态特征语义相似性的新闻类短视频关键帧摘要模型。与传统融合模型不同，该模型在提取多模态特征的基础上，构建公共语义空间，通过最小化对比损失函数对图像-文本对进行联合训练，实现音频文本摘要与视频帧之间语义相似性的跨模态度量，在摘要生成任务中重点关注与音频中语义信息描述一致的图像内容，利用音频中的语义信息筛选相关关键帧，得到更准确的短视频摘要。采集450条CCTV新闻短视频和385条Bilibili自媒体新闻短视频组成实验数据集，使用F1值衡量不同模型的性能，实验结果表明，该模型在2个数据集上F1值分别达到62.8%和51.2%，相较于MSVA模型分别提升了2.1和0.8个百分点，在新闻类短视频关键帧摘要任务中具有更好的性能。

关键词: 声画关系, 主声说, 多模态特征, 语义相似性, 关键帧摘要

Abstract:

According to the "sound and picture relationship" theory of communication, news short videos can directly and effectively convey the video content through audio, which belong to a typical voice-dominated video. Existing video summarization technologies ignore the influence of sound and picture relationships on the performance of video content, resulting in an unstable performance for specific types of short video summarization. Aiming at the characteristics of "voice-dominated" news short videos, this paper proposes a Key Frame Summarization model for News Short Video(KFS4NSV)based on the multimodal features semantic similarity. In contrast to the traditional fusion model, which is based on extracting multimodal features, this model constructs a common semantic space and jointly trains image-text pairs by minimizing the contrast loss function to achieve the cross-modal semantic similarity metric between audio text summarization and video frames. In the summarization generation task, the model focuses on image content consistent with the semantic information in the audio and uses the semantic information in the audio to filter relevant key frames and obtain a more accurate short video summarization. The experimental datasets consisted of 450 short CCTV news videos and 385 short Bilibili self-media news videos. The F1 value is introduced to measure the performance of different models, and the experimental results show that the F1 values of the proposed model on two datasets reach 62.8% and 51.2%, respectively, which are 2.1 and 0.8 percentage points higher, respectively, than those obtained using the MSVA model. The proposed model exhibits superior performance in the news short video key frame summarization task.

Key words: sound and picture relationship, voice-dominated theory, multimodal feature, semantic similarity, key frame summarization

崔晓丹, 刘达维, 刘逸凡, 赵志滨, 任酉贵, 闫永明. 新闻类短视频关键帧摘要模型的研究与实现[J]. 计算机工程, 2023, 49(8): 182-189.

Xiaodan CUI, Dawei LIU, Yifan LIU, Zhibin ZHAO, Yougui REN, Yongming YAN. Research and Implementation of Key Frame Summarization Model for News Short Video[J]. Computer Engineering, 2023, 49(8): 182-189.

https://www.ecice06.com/CN/Y2023/V49/I8/182

图/表 7

图1 KFS4NSV模型结构

Fig.1 KFS4NSV model structure

图2 基于BERT的文本特征提取

Fig.2 Text feature extraction based on BERT

图3 基于VGG19的图像特征提取

Fig.3 Image feature extraction based on VGG19

图4

$ \boldsymbol{\delta } $

值对模型性能的影响

Fig.4 Effect of

$ \boldsymbol{\delta } $

value on the model performance

表1 不同模型的性能比较

Table 1 Performance comparison of different models %

模型	CCTV			Bilibili
模型	P	R	F₁	P	R	F₁
vsLSTM	53.9	53.4	53.6	37.2	36.4	36.8
dppLSTM	54.8	53.6	54.2	38.7	37.6	38.1
SUM-GAN_sup	55.3	56.2	55.7	40.6	41.9	41.2
CSNet_sup	57.6	59.3	58.4	47.6	49.1	48.3
SASUM_sup	58.8	57.9	58.3	44.8	46.5	45.6
MSVA	61.4	60.1	60.7	49.7	51.1	50.4
AVRN	61.0	59.7	60.3	47.2	46.3	46.7
KFS4NSV	62.4	63.3	62.8	51.9	50.6	51.2

表2 2个数据集上消融实验结果

Table 2 Results of ablation experiment on two datasets %

实验编号	F	W	S	CCTV	Bilibili
1	√	×	×	53.2	36.5
2	√	√	×	55.4	41.8
3	√	√	√	62.8	51.2

图5 KFS4NSV模型关键帧摘要示例

Fig.5 Key frame summarization example of KFS4NSV model

参考文献 27

1	朱光烈. "声画结合"论批判(下)──兼做对《对一个定论的异议》批评的回应. 现代传播-北京广播学院学报, 1999, 21(5): 89- 97. URL
	ZHU G L. Criticism on the theory of "combination of sound and picture"(part two): concurrently responding to the criticism of objection to a conclusion. Modern Communication-Journal of Beijing Broadcasting University, 1999, 21(5): 89- 97. URL
2	何国平. 改革开放以来电视新闻节目形态的演化及其声画关系. 中国广播电视学刊, 2009,(2): 30- 31. URL
	HE G P. Evolution of TV news program form and its relationship between sound and picture since reform and opening up. China Radio & TV Academic Journal, 2009,(2): 30- 31. URL
3	朱羽君. 屏幕上的革命——在舟山"电视声画关系"研讨会上的发言[M]//洪民生. 电视声画论集. 北京: 人民出版社, 1993: 306.
	ZHU Y J. The revolution on the screen: speech at the Zhoushan seminar on "the relationship between TV sound and picture"[M]//HONG M S. Anthology of TV sound and picture. Beijing: People's Publishing House, 1993: 306. (in Chinese)
4	RAV-ACHA A, PRITCH Y, PELEG S. Making a long video short: dynamic video synopsis[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2006: 435-441.
5	ZHANG Q, YU S P, ZHOU D S, et al. An efficient method of key-frame extraction based on a cluster algorithm. Journal of Human Kinetics, 2013, 39, 5- 13. doi: 10.2478/hukin-2013-0063
6	AVILA S E F, LOPES A P B, DA LUZ A, et al. VSUMM: a mechanism designed to produce static video summaries and a novel evaluation method. Pattern Recognition Letters, 2011, 32(1): 56- 68. doi: 10.1016/j.patrec.2010.08.004
7	BELO L, CAETANO C, PATROCÍNIO Z, et al. Graph-based hierarchical video summarization using global descriptors[C]//Proceedings of the 26th International Conference on Tools with Artificial Intelligence. Washington D. C., USA: IEEE Press, 2014: 822-829.
8	LIN J X, ZHONG S H, FARES A. Deep hierarchical LSTM networks with attention for video summarization. Computers & Electrical Engineering, 2022, 97, 107618.
9	JI Z, XIONG K L, PANG Y W, et al. Video summarization with attention-based encoder-decoder networks. IEEE Transactions on Circuits and Systems for Video Technology, 2020, 30(6): 1709- 1717. doi: 10.1109/TCSVT.2019.2904996
10	ZHU W C, LU J W, LI J H, et al. DSNet: a flexible detect-to-summarize network for video summarization. IEEE Transactions on Image Processing, 2021, 30, 948- 962. doi: 10.1109/TIP.2020.3039886
11	FAJTL J, SOKEH H S, ARGYRIOU V, et al. Summarizing videos with attention[C]//Proceedings of Asian Conference on Computer Vision. Berlin, Germany: Springer, 2019: 39-54.
12	ZHANG K, CHAO W L, SHA F, et al. Video summarization with long short-term memory[C]//Proceedings of European Conference on Computer Vision. Berlin, Germany: Springer, 2016: 766-782.
13	MAHASSENI B, LAM M, TODOROVIC S. Unsupervised video summarization with adversarial LSTM networks[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2017: 2982-2991.
14	JUNG Y, CHO D, KIM D, et al. Discriminative feature learning for unsupervised video summarization. Proceedings of the AAAI Conference on Artificial Intelligence, 2019, 33(1): 8537- 8544. doi: 10.1609/aaai.v33i01.33018537
15	YAO T, MEI T, RUI Y. Highlight detection with pairwise deep ranking for first-person video summarization[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 982-990.
16	HORI C, HORI T, LEE T Y, et al. Attention-based multimodal fusion for video description[C]//Proceedings of IEEE International Conference on Computer Vision. Washington D. C., USA: IEEE Press, 2017: 4203-4212.
17	WEI H W, NI B B, YAN Y C, et al. Video summarization via semantic attended networks. Proceedings of the AAAI Conference on Artificial Intelligence, 2018, 32(1): 216- 223.
18	GHAURI J A, HAKIMOV S, EWERTH R. Supervised video summarization via multiple feature sets with parallel attention[C]//Proceedings of IEEE International Conference on Multimedia and Expo. Washington D. C., USA: IEEE Press, 2021: 1-6.
19	ZHAO B, GONG M G, LI X L. AudioVisual video summarization[EB/OL]. [2022-08-10]. https://arxiv.org/abs/2105.07667v1.
20	OTANI M, NAKASHIMA Y, RAHTU E, et al. Video summarization using deep semantic features[C]//Proceedings of Asian Conference on Computer Vision. Berlin, Germany: Springer, 2017: 361-377.
21	LI Y B, MERIALDO B. Multi-video summarization based on AV-MMR[C]//Proceedings of International Workshop on Content Based Multimedia Indexing. Washington D. C., USA: IEEE Press, 2010: 1-6.
22	YAO Y F. Semantic feature hierarchical clustering algorithm based on improved regional merging strategy. Cluster Computing, 2019, 22(1): 1495- 1503.
23	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional Transformers for language understanding[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1810.04805.
24	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1409.1556.
25	CHOPRA S, HADSELL R, LECUN Y. Learning a similarity metric discriminatively, with application to face verification[C]//Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2005: 539-546.
26	XU J, MEI T, YAO T, et al. MSR-VTT: a large video description dataset for bridging video and language[C]//Proceedings of IEEE Conference on Computer Vision and Pattern Recognition. Washington D. C., USA: IEEE Press, 2016: 5288-5296.
27	KINGMA D P, BA J. Adam: a method for stochastic optimization[EB/OL]. [2022-08-10]. https://arxiv.org/abs/1412.6980.

[1]	林梦琪, 张晓梅. 基于行为足迹的多模态融合身份认证[J]. 计算机工程, 2021, 47(10): 116-124.
[2]	吴玺煜,陈启买,刘海,贺超波. 基于知识图谱表示学习的协同过滤推荐算法[J]. 计算机工程, 2018, 44(2): 226-232,263.
[3]	罗永恩,胡继承,徐茜. 基于超图的多模态关联特征处理方法[J]. 计算机工程, 2017, 43(1): 226-230.
[4]	曾剑平,刘华. 一种基于聚集系数的人名识别方法[J]. 计算机工程, 2016, 42(7): 203-208.
[5]	刘嘉琦, 封化民, 闫建鹏. 基于多模态特征融合的新闻故事单元分割[J]. 计算机工程, 2012, 38(24): 161-165.

选择文件类型/文献管理软件名称

选择包含的内容

新闻类短视频关键帧摘要模型的研究与实现

Research and Implementation of Key Frame Summarization Model for News Short Video

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 27

相关文章 5

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

新闻类短视频关键帧摘要模型的研究与实现

Research and Implementation of Key Frame Summarization Model for News Short Video

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 27

相关文章 5

编辑推荐

Metrics

本文评价