Support and Optimization of Multi-Granularity Quantization Framework for Deep Learning Compiler

doi:10.19678/j.issn.1000-3428.0069206

Abstract

Abstract:

With the increasing demand for the deployment of large models by major manufacturers, the accuracy of the single quantization method of deep learning compiler Tensor Virtual Machine (TVM) has decreased, and this method is no longer sufficient to satisfy deployment requirements. Therefore, in this study, a flexible granularity model quantization framework is designed and constructed. This framework supports layer-wise and channel-wise quantization processes as well as the implementation of threshold search and adaptive rounding optimization algorithms. First, based on the quantization module ″relay.quantize″, a framework flow for information annotation, threshold calibration, and quantization graph realization is constructed, which includes granularity attributes to explicitly identify the quantization method. Second, fine-tuning is applied to the threshold calibration and weight rounding in quantization to address the issue of ineffective quantization information determination using predefined calibration methods, thereby improving the accuracy of the quantized model. Experiments are conducted using the ImageNet dataset to test visual networks. The results reveal that the new quantization scheme for MobileNetV1 reduces the loss of model accuracy to 2.3% after 8 bit quantization, and this loss is reduced to 0.7% after tuning. Hence, the multi-granularity quantization framework can effectively reduce the quantization error.

Key words: model quantization, model deployment, model compression, inference acceleration, deep learning compiler

摘要：

随着各大厂商对大模型应用部署需求的激增, 深度学习编译器TVM(Tensor Virtual Machine)的单一量化方式精度下降, 已无法满足部署需求。设计并构建一种可选粒度的模型量化框架, 具体包括逐层与逐通道量化流程的支持, 以及阈值搜索与自适应舍入优化算法的实现。首先, 基于量化模块“relay.quantize”构建信息标注、阈值校准与量化图实现的框架流程, 并添加粒度属性以显式识别量化方式。其次, 针对预定义校准方法无法确定有效量化信息的问题, 对量化中的阈值校准、权重舍入进行调优, 提高量化后模型精度。实验采用ImageNet数据集对视觉网络进行测试, 针对MobileNetV1新量化方案将8 bit量化后模型精度损失降低到2.3%, 调优后该损失降低到0.7%, 实验结果表明多粒度量化框架可有效降低量化误差。

关键词: 模型量化, 模型部署, 模型压缩, 推理加速, 深度学习编译器

WEI Mingkang, LI Jianan, HAN Lin, GAO Wei, ZHAO Rongcai, WANG Hongsheng. Support and Optimization of Multi-Granularity Quantization Framework for Deep Learning Compiler[J]. Computer Engineering, 2025, 51(5): 62-72.

魏铭康, 李嘉楠, 韩林, 高伟, 赵荣彩, 王洪生. 面向深度学习编译器的多粒度量化框架支持与优化[J]. 计算机工程, 2025, 51(5): 62-72.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0069206

https://www.ecice06.com/EN/Y2025/V51/I5/62

Figures/Tables 18

Fig.1 Example of quantization and dequantization

Fig.2 Example of quantization granularity

Fig.3 Comparison of quantization frameworks

Fig.4 Quantization process

Fig.5 Example of quantization granularity properties

Fig.6 Example of information annoatation for various operators

Fig.7 Example of computation graph reconstruction in threshold calibration

Fig.8 Example of quantization graph realization for various operators

Fig.9 Relationship between threshold selection and accuracy loss

Fig.10 Example of quantize operator transformation

Fig.11 Model compression ratio

Fig.12 Speed-up ratio of various models under different quantization granularities

Fig.13 Speed-up ratio compared to MXNet

Fig.14 Speed-up ratio compared to PyTorch

Fig.15 Relationship between the number of fine-tuning layers and accuracy

References 25

1	PÉREZ-GUERRERO C , CIPRIÁN-SÁNCHEZ J F , PALACIOS A , et al. Computer vision-based characterization of large-scale jet flames using a synthetic infrared image generation approach. Engineering Applications of Artificial Intelligence, 2024, 127, 107275. doi: 10.1016/j.engappai.2023.107275
2	EYIOKUR F I , KANTARCI A , ERAKıN M E , et al. A survey on computer vision based human analysis in the COVID-19 era. Image and Vision Computing, 2023, 130, 104610. doi: 10.1016/j.imavis.2022.104610
3	KHURANA D , KOLI A , KHATTER K , et al. Natural language processing: state of the art, current trends and challenges. Multimedia Tools and Applications, 2023, 82 (3): 3713- 3744. doi: 10.1007/s11042-022-13428-4
4	LIAO J W , ESKIMEZ S , LU L Y , et al. Improving readability for automatic speech recognition transcription. ACM Transactions on Asian and Low-Resource Language Information Processing, 2023, 22 (5): 1- 23.
5	LIU F Y, BUGLIARELLO E, PONTI E M, et al. Visually grounded reasoning across languages and cultures[EB/OL]. [2024-01-09]. https://arxiv.org/abs/2109.13238v2.
6	王军, 冯孙铖, 程勇. 深度学习的轻量化神经网络结构研究综述. 计算机工程, 2021, 47 (8): 1- 13. URL
	WANG J , FENG S C , CHENG Y . Survey of research on lightweight neural network structures for deep learning. Computer Engineering, 2021, 47 (8): 1- 13. URL
7	OGBOGU C O , ARKA A I , PFROMM L , et al. Accelerating graph neural network training on ReRAM-based PIM architectures via graph and model pruning. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2023, 42 (8): 2703- 2716. doi: 10.1109/TCAD.2022.3227879
8	STEWART J, MICHIELI U, OZAY M. Data-free model pruning at initialization via expanders[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2023: 4518-4523.
9	郭奕裕, 周箩鱼. 安全帽佩戴检测网络模型的轻量化设计. 计算机工程, 2023, 49 (4): 312- 320. URL
	GUO Y Y , ZHOU L Y . Lightweight design of safety helmet wearing detection network model. Computer Engineering, 2023, 49 (4): 312- 320. URL
10	GOU J P , XIONG X S , YU B S , et al. Multi-target knowledge distillation via student self-reflection. International Journal of Computer Vision, 2023, 131 (7): 1857- 1874. doi: 10.1007/s11263-023-01792-z
11	曹坪, 杨怀志, 薄一军, 等. 面向低质量裂缝图像的多知识蒸馏分类. 计算机工程, 2023, 49 (7): 204- 213. URL
	CAO P , YANG H Z , BO Y J , et al. Low-quality crack image classification with multi-knowledge distillation. Computer Engineering, 2023, 49 (7): 204- 213. URL
12	ZOU H , ZHANG C , LASAULCE S , et al. Goal-oriented quantization: analysis, design, and application to resource allocation. IEEE Journal on Selected Areas in Communications, 2022, 41 (1): 42- 54.
13	XU N J , CHEN X H , CAO Y L , et al. Hybrid post-training quantization for super-resolution neural network compression. IEEE Signal Processing Letters, 2023, 30, 379- 383.
14	巩杰, 赵烁, 何虎, 等. 基于FPGA的量化CNN加速系统设计. 计算机工程, 2022, 48 (3): 170-174, 196. URL
	GONG J , ZHAO S , HE H , et al. Design of quantized CNN acceleration system based on FPGA. Computer Engineering, 2022, 48 (3): 170-174, 196. URL
15	CHEN T Q, MOREAU T, JIANG Z H, et al. TVM[C]//Proceedings of the 13th USENIX Conference on Operating Systems Design and Implementation. New York, USA: ACM Press, 2018: 579-594.
16	LI M Z , LIU Y , LIU X Y , et al. The deep learning compiler: a comprehensive survey. IEEE Transactions on Parallel and Distributed Systems, 2020, 32 (3): 708- 727.
17	ZHANG H, XING M, WU Y, et al. Compiler technologies in deep learning co-design: a survey[EB/OL]. [2024-01-09]. https://spj.science.org/doi/10.34133/icomputing.0040.
18	ROESCH J, LYUBOMIRSKY S, WEBER L, et al. Relay: a new IR for machine learning frameworks[C]//Proceedings of the 2nd ACM SIGPLAN International Workshop on Machine Learning and Programming Languages. New York, USA: ACM Press, 2018: 58-68.
19	JAIN A, BHATTACHARYA S, MASUDA M, et al. Efficient execution of quantized deep learning models: a compiler approach[EB/OL]. (2020-06-18)[2024-01-09]. https://doi.org/10.48550/arXiv.2006.10226.
20	KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, 60 (6): 84- 90.
21	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 770-778.
22	SZEGEDY C, VANHOUCKE V, IOFFE S, et al. Rethinking the inception architecture for computer vision[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2016: 2818-2826.
23	SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL]. (2015-04-10)[2024-01-09]. https://doi.org/10.48550/arXiv.1409.1556.
24	SANDLER M, HOWARD A, ZHU M L, et al. MobileNetV2: inverted residuals and linear bottlenecks[C]//Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). Washington D. C., USA: IEEE Press, 2018: 4510-4520.
25	CHEN T, LI M, LI Y, et al. MXNet: a flexible and efficient machine learning library for heterogeneous distributed systems[EB/OL]. (2015-12-03)[2024-01-09]. https://doi.org/10.48550/arXiv.1512.01274.

[1]	WANG Yuehao, ZHOU Ruohua. Review of Research on Keyword Spotting in Low-Resource Environments [J]. Computer Engineering, 2025, 51(2): 35-53.
[2]	Wei GAO, Shuailong LI, Lin MAO, Lei WANG, Yingying LI, Lin HAN. One Acceleration Strategy for Operator Generation Based on TVM [J]. Computer Engineering, 2024, 50(8): 353-362.
[3]	LI Yiting, QU Dan, YANG Xukui, ZHANG Hao, SHEN Xiaolong. Efficient Conformer Model Based on Factorized Gated Attention Unit [J]. Computer Engineering, 2023, 49(5): 73-80.
[4]	GUO Yiyu, ZHOU Luoyu. Lightweight Design of Safety Helmet Wearing Detection Network Model [J]. Computer Engineering, 2023, 49(4): 312-320.
[5]	WANG Guodong, YE Jian, XIE Ying, QIAN Yueliang. Structured Pruning Algorithm with Adaptive Threshold Based on Gradient [J]. Computer Engineering, 2022, 48(9): 113-120.
[6]	WANG Shihao, WANG Zhongqing, LI Shoushan, ZHOU Guodong. Event Argument Extraction Method Based on Knowledge Distillation and Model Ensemble [J]. Computer Engineering, 2022, 48(7): 97-103.
[7]	KONG Weigang, LI Wenjing, WANG Qiuyan, CAO Pengcheng, SONG Qingzeng. Design and Implementation of Lightweight Network Based on Improved YOLOv4 Algorithm [J]. Computer Engineering, 2022, 48(3): 181-188.
[8]	GONG Jie, ZHAO Shuo, HE Hu, DENG Ning. Design of Quantized CNN Acceleration System Based on FPGA [J]. Computer Engineering, 2022, 48(3): 170-174,196.
[9]	LU Peng, WAN Ying, ZOU Guoliang, CHEN Jinyu, ZHENG Zongsheng, WANG Zhenhua. Neural Network Model Compression Based on Adaptive Hierarchical Threshold Judgment [J]. Computer Engineering, 2022, 48(1): 112-118,126.
[10]	HUANG Rui, JIN Guanghao, LI Lei, JIANG Wenchao, SONG Qingzeng. Design and Implementation of Accelerator for Lightweight Neural Network [J]. Computer Engineering, 2021, 47(9): 185-190,196.
[11]	WANG Jun, FENG Suncheng, CHENG Yong. Survey of Research on Lightweight Neural Network Structures for Deep Learning [J]. Computer Engineering, 2021, 47(8): 1-13.
[12]	ZHANG Hongmei, YAN Haibing, ZHANG Xiangli. Neural Network Compression Method Combining Half-Wave Gaussian Quantization and Alternate Update [J]. Computer Engineering, 2021, 47(5): 80-87.
[13]	YANG Minjie, LIANG Yaling, DU Minghui. YOLO Pruning Algorithm Based on Parameter Subspace and Scaling Factor [J]. Computer Engineering, 2021, 47(2): 111-117.
[14]	WEI Yue, CHEN Shichao, ZHU Fenghua, XIONG Gang. Pruning Method for Convolutional Neural Network Models Based on Sparse Regularization [J]. Computer Engineering, 2021, 47(10): 61-66.
[15]	LIU Xuan, CHI Mingmin. Scene Classification Research for Remote Sensing Images Based on Multi?Granularity Feature Distillation [J]. Computer Engineering, 2021, 47(1): 224-229,238.

Please choose a citation manager

Content to export