基于可变序列的跨层通道剪枝方法

doi:10.19678/j.issn.1000-3428.0069751

计算机工程 ›› 2026, Vol. 52 ›› Issue (1): 381-389. doi: 10.19678/j.issn.1000-3428.0069751

基于可变序列的跨层通道剪枝方法

李秉烨¹^,², 鲍宇¹^,²^,*(), 曹伟³, 李明泽³, 徐方正¹^,²

1. 中国矿业大学计算机科学与技术学院/人工智能学院, 江苏徐州 221116
2. 中国矿业大学矿山数字化教育部工程研究中心, 江苏徐州 221116
3. 徐州市第一人民医院, 江苏徐州 221116

收稿日期:2024-04-15 修回日期:2024-07-10 出版日期:2026-01-15 发布日期:2024-10-18
通讯作者: 鲍宇
作者简介:
李秉烨, 男, 硕士研究生, 主研方向为软件工程、人工智能
鲍宇(CCF专业会员、通信作者), 副教授、博士
曹伟, 主任医师
李明泽, 住院医师
徐方正, 硕士研究生
基金资助:
国家自然科学基金(52374164); 徐州市医药卫生面上研究项目(KC22110)

Cross-layer Channel Pruning Method Based on Variable Sequences

LI Bingye¹^,², BAO Yu¹^,²^,*(), CAO Wei³, LI Mingze³, XU Fangzheng¹^,²

1. School of Computer Science and Technology/School of Artificial Intelligence, China University of Mining and Technology, Xuzhou 221116, Jiangsu, China
2. Mine Digitization Engineering Research Center of the Ministry of Education, China University of Mining and Technology, Xuzhou 221116, Jiangsu, China
3. Xuzhou No. 1 People's Hospital, Xuzhou 221116, Jiangsu, China

Received:2024-04-15 Revised:2024-07-10 Online:2026-01-15 Published:2024-10-18
Contact: BAO Yu

摘要/Abstract

摘要：

放射治疗是肝癌的重要治疗方式。基于深度学习的图像语义分割技术能辅助医师勾画放射靶区, 提高放射治疗的精确性。然而, 现有的医疗图像语义分割模型结构较为复杂, 且参数量大, 难以部署在资源受限的设备上。分析视觉Transformer模型参数的重要性, 发现模型不同层次的重要性参数具有特殊分布规律。据此, 提出基于可变序列的跨层通道剪枝方法。依据重要性参数分布规律, 对多头自注意力(MSA)层和前馈网络(FFN)层, 测量其重要性权重并调节取值, 形成重要性权重值的层次序列, 再为序列设定对应的剪枝率, 形成随网络深度变化的可变剪枝率序列, 从而实现MSA层和FFN层的精细化剪枝。引入循环剪枝策略, 在每一轮次模型剪枝过程中迭代更新可变剪枝率序列, 以充分削减MSA层和FFN层的冗余结构。在公开肝脏分割数据集3D-IRCADb-01上进行训练和测试, 视觉Transformer模型经剪枝后, 图像分割准确率保持不变, 但浮点运算数(FLOPs)和参数量分别减少了60.26%和66.07%。实验结果表明, 所提方法在保证分割精度的前提下, 实现了更大的剪枝率, 比固定剪枝率方法更具有优势。

关键词: 医学影像, 语义分割, 视觉Transformer, 可变序列, 模型剪枝

Abstract:

Radiotherapy is an important treatment modality for liver cancer. Deep learning-based image semantic segmentation technology can assist physicians in demarcating radiation target areas and enhancing the accuracy of radiotherapy. However, existing medical image semantic segmentation models are relatively intricate and possess a substantial number of parameters, rendering them challenging to deploy on devices with constrained resources. An analysis of the significance of the parameters of the vision transformer model reveals that the crucial parameters of the different layers of the model exhibit a distinctive distribution pattern. Based on this finding, this study proposes a cross-layer channel pruning method based on variable sequences. According to the distribution pattern of the significant parameters, the significance weights of the Multihead Self-Attention (MSA) and Feed-Forward Network (FFN) layers are measured and these values are adjusted to form a hierarchical sequence of significance weight values. Subsequently, the corresponding pruning rate is set for the sequence to form a variable pruning rate sequence that varies with the depth of the network, thereby achieving fine pruning of the MSA and FFN layers. This new method introduces a cyclic pruning strategy that iteratively updates the variable pruning rate sequence during each round of model pruning to reduce the redundant structures in the MSA and FFN layers adequately. The model is trained and tested using the public liver segmentation dataset, 3D-IRCADb-01. After pruning the vision transformer, the accuracy of the image segmentation does not decline and the Floating-Point Operations (FLOPs) and number of parameters are reduced by 60.26% and 66.07%, respectively. Experimental results indicate that the new method attains a higher pruning rate while guaranteeing segmentation accuracy and is more advantageous than the fixed pruning rate method.

Key words: medical imaging, semantic segmentation, vision Transformer, variable sequence, model pruning

李秉烨, 鲍宇, 曹伟, 李明泽, 徐方正. 基于可变序列的跨层通道剪枝方法[J]. 计算机工程, 2026, 52(1): 381-389.

LI Bingye, BAO Yu, CAO Wei, LI Mingze, XU Fangzheng. Cross-layer Channel Pruning Method Based on Variable Sequences[J]. Computer Engineering, 2026, 52(1): 381-389.

https://www.ecice06.com/CN/Y2026/V52/I1/381

图/表 7

图1 原生视觉Transfomer模型结构

Fig.1 Native vision Transformer model structure

图2 不同深度网络层通道剪枝率

Fig.2 Channel pruning rates in each network layer

图3 模型剪枝方法示意图

Fig.3 Schematic diagram of the model pruning method

图4 模型分割效果

Fig.4 Segmentation effect of models

参考文献 24

1	YANG Z J, XIE L X, ZHOU W, et al. VoxSeP: semi-positive voxels assist self-supervised 3D medical segmentation. Multimedia Systems, 2023, 29(1): 33- 48. doi: 10.1007/s00530-022-00977-9
2	NARAYAN V, FAIZ M, MALL P K, et al. A comprehensive review of various approach for medical image segmentation and disease prediction. Wireless Personal Communications, 2023, 132(3): 1819- 1848. doi: 10.1007/s11277-023-10682-z
3	ISAKSSON L J, SUMMERS P, MASTROLEO F, et al. Automatic segmentation with deep learning in radiotherapy. Cancers, 2023, 15(17): 4389. doi: 10.3390/cancers15174389
4	LIANG B T, TANG C, ZHANG W, et al. N-Net: an UNet architecture with dual encoder for medical image segmentation. Signal, Image and Video Processing, 2023, 17(6): 3073- 3081. doi: 10.1007/s11760-023-02528-9
5	BOUGOURZI F, DISTANTE C, DORNAIKA F, et al. PDAtt-Unet: pyramid dual-decoder attention Unet for COVID-19 infection segmentation from CT-scans. Medical Image Analysis, 2023, 86, 102797. doi: 10.1016/j.media.2023.102797
6	WU H S, ZHAO Z B, WANG Z Z. META-Unet: multi-scale efficient Transformer attention unet for fast and high-accuracy polyp segmentation. IEEE Transactions on Automation Science and Engineering, 2024, 21(3): 4117- 4128. doi: 10.1109/TASE.2023.3292373
7	VALANARASU J M J, PATEL V M. UNeXt: MLP-based rapid medical image segmentation network. Berlin, Germany: Springer, 2022: 23- 33.
8	HAN W Z, ZHANG Y T, ZHAO Y C, et al. 3D U-Net3 based microbubble filtering for ultrasound localization microscopy[C]//Proceedings of the IEEE International Conference on Systems, Man, and Cybernetics (SMC). Washington D.C., USA: IEEE Press, 2023: 3974-3979.
9	LIN A L, CHEN B Z, XU J Y, et al. DS-TransUNet: dual swin Transformer U-Net for medical image segmentation. IEEE Transactions on Instrumentation and Measurement, 2022, 71, 4005615.
10	CAO H, WANG Y Y, CHEN J, et al. Swin-Unet: Unet-like pure Transformer for medical image segmentation. Belrin, Germany: Springer, 2023.
11	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. New York, USA: ACM Press, 2017: 6000-6010.
12	XIE E Z, WANG W H, YU Z D, et al. SegFormer: simple and efficient design for semantic segmentation with Transformers. Advances in Neural Information Processing Systems, 2021, 34, 12077- 12090.
13	HOWARD A, SANDLER M, CHEN B, et al. Searching for MobileNetV3[C]//Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2019: 1314-1324.
14	CHAVAN A, SHEN Z Q, LIU Z, et al. Vision Transformer slimming: multi-dimension searching in continuous optimization space[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2022: 4921-4931.
15	ZAGORUYKO S, KOMODAKIS N. Paying more attention to attention: improving the performance of convolutional neural networks via attention transfer[EB/OL]. [2024-03-11]. https://doi.org/10.48550/arXiv.1612.03928.
16	张帆, 黄赟, 方子茁, 等. 卷积神经网络的损失最小训练后参数量化方法. 通信学报, 2022, 43(4): 114- 122.
	ZHANG F, HUANG Y, FANG Z Z, et al. Lost-minimum post-training parameter quantization method for convolutional neural network. Journal on Communications, 2022, 43(4): 114- 122.
17	GHOLAMI A, KWON K, WU B C, et al. SqueezeNext: hardware-aware neural network design[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW). Washington D.C., USA: IEEE Press, 2018: 1719-171909.
18	王宪保, 刘鹏飞, 项圣, 等. 基于神经架构搜索的非结构化剪枝方法. 模式识别与人工智能, 2023, 36(5): 448- 458.
	WANG X B, LIU P F, XIANG S, et al. Unstructured pruning method based on neural architecture search. Pattern Recognition and Artificial Intelligence, 2023, 36(5): 448- 458.
19	李屹, 魏建国, 刘贯伟. 模型剪枝算法综述. 计算机与现代化, 2022(9): 51- 59.
	LI Y, WEI J G, LIU G W. Survey of model pruning algorithms. Computer and Modernization, 2022(9): 51- 59.
20	LI H, KADAV A, DURDANOVIC I, et al. Pruning filters for efficient ConvNets[EB/OL]. [2024-03-11]. https://arxiv.org/abs/1608.08710.
21	HE Y, LIU P, WANG Z W, et al. Filter pruning via geometric Median for deep convolutional neural networks acceleration[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2019: 1-15.
22	LIU Z, LI J G, SHEN Z Q, et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the IEEE International Conference on Computer Vision (ICCV). Washington D.C., USA: IEEE Press, 2017: 2755-2763.
23	ZHU M, TANG Y, HAN K. Vision Transformer pruning[EB/OL]. [2024-03-11]. https://arxiv.org/abs/2104.08500.
24	ZHENG S X, LU J C, ZHAO H S, et al. Rethinking semantic segmentation from a sequence-to-sequence perspective with Transformers[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2021: 6877-6886.

[1]	刘杰, 黄晓辉, 郭敬博. 基于YOLOv8的轻量级田间棉花品级检测[J]. 计算机工程, 2026, 52(1): 400-413.
[2]	黎东丰, 陈雨人, 余博. 基于多层次特征融合的路面裂缝检测方法[J]. 计算机工程, 2026, 52(1): 154-165.
[3]	杨萍, 张汐. 改进DeepLabv3+的道路表面裂缝检测方法[J]. 计算机工程, 2025, 51(4): 261-270.
[4]	韩鹏, 黄韫栀, 任彩月, 程竞仪, 徐军. 基于双分支网络的乳腺PET新辅助化疗疗效评估[J]. 计算机工程, 2025, 51(3): 293-299.
[5]	雷赛月, 方立, 李辰德, 杨铭. 改进光学卫星图像中表碛覆盖型冰川区域提取算法[J]. 计算机工程, 2025, 51(2): 269-277.
[6]	刘洲峰, 李冰芮, 杨瑞敏, 李春雷, 何媛, 丁淑敏. 基于调制-全局推理的弱监督语义分割算法研究[J]. 计算机工程, 2025, 51(2): 344-355.
[7]	阳佩珉, 闵华松. 嵌入式肌电腕带实时采集与识别系统设计[J]. 计算机工程, 2025, 51(2): 259-268.
[8]	王杨, 宋世佳, 王鹤琴, 袁振羽, 赵立军, 吴其林. 基于改进Vision Transformer的局部光照一致性估计[J]. 计算机工程, 2025, 51(2): 312-321.
[9]	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法[J]. 计算机工程, 2024, 50(8): 282-289.
[10]	李仲, 冒睿瑞, 王晓龙, 王根一, 安国成. 基于改进PIDNet的水位线检测算法[J]. 计算机工程, 2024, 50(8): 102-112.
[11]	肖慈, 徐杨, 张永丹, 冯明文, 黄易仟. 结合注意力和低光增强的夜间语义分割[J]. 计算机工程, 2024, 50(7): 271-281.
[12]	逯焕宇, 张永宏, 马光义, 谢东林, 田伟. 基于半监督对抗学习的遥感图像水体提取[J]. 计算机工程, 2024, 50(7): 251-263.
[13]	陈晓玉, 沈晨, 沈阅, 孔德明. 基于改进SwiftNet的堆场图像实时分割网络[J]. 计算机工程, 2024, 50(6): 296-303.
[14]	王安政, 党建武, 岳彪, 杨景玉. 基于位置信息和注意力机制的路面裂缝检测[J]. 计算机工程, 2024, 50(4): 303-312.
[15]	刘瑞康, 刘伟铭, 段梦飞, 谢玮, 戴愿. 基于双通道Transformer的地铁站台异物检测[J]. 计算机工程, 2024, 50(4): 197-207.

选择文件类型/文献管理软件名称

选择包含的内容

基于可变序列的跨层通道剪枝方法

Cross-layer Channel Pruning Method Based on Variable Sequences

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 24

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于可变序列的跨层通道剪枝方法

Cross-layer Channel Pruning Method Based on Variable Sequences

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 7

参考文献 24

相关文章 15

编辑推荐

Metrics

本文评价