融合最大池化的Conformer中文语音识别

doi:10.19678/j.issn.1000-3428.0070055

计算机工程 ›› 2026, Vol. 52 ›› Issue (1): 105-115. doi: 10.19678/j.issn.1000-3428.0070055

融合最大池化的Conformer中文语音识别

胡从刚¹^,², 杨立鹏³, 孙永奇¹^,²^,*(), 陈华龙³, 韩可可³

1. 北京交通大学先进轨道交通自主运行全国重点实验室, 北京 100044
2. 北京交通大学计算机科学与技术学院, 北京 100044
3. 中国铁道科学研究院集团有限公司, 北京 100081

收稿日期:2024-07-01 修回日期:2024-08-09 出版日期:2026-01-15 发布日期:2026-01-15
通讯作者: 孙永奇
作者简介:
胡从刚(CCF学生会员), 男, 硕士, 主研方向为语音识别、人工智能
杨立鹏, 副研究员、博士
孙永奇(通信作者), 教授、博士、博士生导师
陈华龙, 助理研究员
韩可可, 助理研究员
基金资助:
中央高校基本科研业务费专项资金(2024JBGP008); 新一代人工智能国家科技重大专项(2021ZD0113002)

Chinese Speech Recognition Using Conformer Fused with Max Pooling

HU Conggang¹^,², YANG Lipeng³, SUN Yongqi¹^,²^,*(), CHEN Hualong³, HAN Keke³

1. State Key Laboratory of Advanced Rail Autonomous Operation, Beijing Jiaotong University, Beijing 100044, China
2. School of Computer Science and Technology, Beijing Jiaotong University, Beijing 100044, China
3. China Academy of Railway Sciences Group Co., Ltd., Beijing 100081, China

Received:2024-07-01 Revised:2024-08-09 Online:2026-01-15 Published:2026-01-15
Contact: SUN Yongqi

摘要/Abstract

摘要：

语音识别旨在通过先进的算法与信号处理技术, 赋予机器理解人类语音的能力, 使得人与机器之间的交流更加便捷、顺畅。目前, 大多数端到端语音识别的研究工作主要围绕Conformer模型进行优化。针对Conformer编码器对语音细粒度局部特征提取能力不足的问题, 提出一种融合最大池化(MP)的Conformer中文语音识别模型。首先, 将编码器卷积模块中门控线性单元的输出在时间维度上进行MP, 以提取多帧语音信号对应一个字符的细粒度局部特征。然后, 将池化后的特征与逐通道卷积(DWC)提取的粗粒度局部特征以逐元素相加的方式进行融合, 以增加语音局部特征的信息量, 从而提高Conformer模型的语音识别准确率。最后, 在公开的中文数据集Aishell-1上的实验结果表明: 采用贪心搜索方式进行解码, 所提模型可以将基线模型的字错误率(CER)从5.58%降低至5.32%;采用注意力重打分方式进行解码, 所提模型可以将基线模型的CER从5.06%降低至4.92%。

关键词: 语音识别, 细粒度局部特征, Conformer模型, 最大池化, 逐通道卷积

Abstract:

Speech recognition technology enables machines to understand human speech using advanced algorithms and signal processing technologies, thereby making communication between humans and machines more convenient. Most existing studies on end-to-end speech recognition focus on optimizing the Conformer model. The Conformer encoder suffers from the issue of insufficient extraction of fine-grained local speech features. To resolve these issues, this study proposes a Chinese speech recognition method based on Max Pooling (MP). First, the output of the gated linear unit in the convolutional module of the encoder is max-pooled along the time dimension to extract fine-grained local features corresponding to the characteristics of multiple speech signal frames. Second, these pooled features are fused with the coarse-grained local features extracted via Depthwise Convolution (DWC) using the element-wise sum method to increase the amount of information on local speech features and improve the speech recognition accuracy of the Conformer model. The experimental results on the public Chinese dataset Aishell-1 show that the improved model can reduce the Character Error Rate (CER) of the baseline model from 5.58% to 5.32% and from 5.06% to 4.92% by decoding using greedy search and attention rescoring, respectively.

Key words: speech recognition, fine-grained local feature, Conformer model, Max Pooling (MP), Depthwise Convolution (DWC)

胡从刚, 杨立鹏, 孙永奇, 陈华龙, 韩可可. 融合最大池化的Conformer中文语音识别[J]. 计算机工程, 2026, 52(1): 105-115.

HU Conggang, YANG Lipeng, SUN Yongqi, CHEN Hualong, HAN Keke. Chinese Speech Recognition Using Conformer Fused with Max Pooling[J]. Computer Engineering, 2026, 52(1): 105-115.

https://www.ecice06.com/CN/Y2026/V52/I1/105

图/表 22

图1 语音特征提取过程

Fig.1 Speech feature extraction process

图2 卷积模块

Fig.2 Convolution module

图3 DWC过程

Fig.3 DWC process

图4 MP-Conformer模型结构

Fig.4 MP-Conformer model architecture

图5 MP & 卷积模块

Fig.5 MP & convolution module

图6 FBank特征图

Fig.6 FBank feature map

图7 编码器的输入特征图

Fig.7 Input feature map of the encoder

图8 一维MP过程

Fig.8 One-dimensional MP process

图9 MP与DWC融合的串行结构

Fig.9 Serial structure fused with MP and DWC

图10 不同模型的验证集损失值对比

Fig.10 Comparison of validation set loss of different models

图11 验证集损失值对比

Fig.11 Comparison of validation set loss

图12 不同尺度的MP-Conformer实验结果对比

Fig.12 Comparison of experimental results of MP-Conformer with different scales

图13 不同比例的MP-Conformer实验结果对比

Fig.13 Comparison of experimental results of MP-Conformer with different ratios

图14 不同解码方式的模型实验结果对比

Fig.14 Comparison of experimental results of models with different decoding methods

参考文献 25

1	洪青阳, 李琳. 语音识别: 原理与应用. 北京: 电子工业出版社, 2020.
	HONG Q Y , LI L . Principle and application of speech recognition. Beijing: Publishing House of Electronics Industry, 2020.
2	DAHL G E , YU D , DENG L , et al. Context-dependent pre-trained deep neural networks for large-vocabulary speech recognition. IEEE Transactions on Audio Speech and Language Processing, 2012, 20 (1): 30- 42. doi: 10.1109/TASL.2011.2134090
3	GRAVES A, MOHAMED A R, HINTON G. Speech recognition with deep recurrent neural networks[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D.C., USA: IEEE Press, 2013: 6645-6649.
4	GRAVES A, FERNÁNDEZ S, GOMEZ F, et al. Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks[C]//Proceedings of the 23rd International Conference on Machine Learning. New York, USA: ACM Press, 2006: 369-376.
5	GRAVES A. Sequence transduction with recurrent neural networks[EB/OL]. [2024-06-08]. https://lorenlugosch.github.io/posts/2020/11/transducer/.
6	CHAN W, JAITLY N, LE Q, et al. Listen, attend and spell: a neural network for large vocabulary conversational speech recognition[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2016: 4960-4964.
7	WANG D , WANG X , LÜ S . An overview of end-to-end automatic speech recognition. Symmetry, 2019, 11 (8): 1018. doi: 10.3390/sym11081018
8	DONG L H, XU S, XU B. Speech-Transformer: a no-recurrence sequence-to-sequence model for speech recognition[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2018: 5884-5888.
9	ZHOU S Y , DONG L H , XU S , et al. A comparison of modeling units in sequence-to-sequence speech recognition with the Transformer on mandarin Chinese. Berlin, Germany: Springer International Publishing, 2018.
10	赵德春, 舒洋, 李玲, 等. 丢弃冗余块的语音识别Transformer解码加速方法. 计算机工程, 2023, 49 (10): 105-111, 119. doi: 10.19678/j.issn.1000-3428.0065685
	ZHAO D C , SHU Y , LI L , et al. Speech recognition Transformer decoding acceleration method with discarding redundant blocks. Computer Engineering, 2023, 49 (10): 105-111, 119. doi: 10.19678/j.issn.1000-3428.0065685
11	GULATI A, QIN J, CHIU C C, et al. Conformer: convolution-augmented Transformer for speech recognition[EB/OL]. [2024-06-08]. https://arxiv.org/abs/2005.08100.
12	BURCHI M, VIELZEUF V. Efficient Conformer: progressive downsampling and grouped attention for automatic speech recognition[C]//Proceedings of the IEEE Automatic Speech Recognition and Understanding Workshop (ASRU). Washington D.C., USA: IEEE Press, 2021: 8-15.
13	GAO Z, ZHANG S, MCLOUGHLIN I, et al. Paraformer: fast and accurate parallel Transformer for non-autoregressive end-to-end speech recognition[EB/OL]. [2024-06-08]. https://arxiv.org/abs/2206.08317.
14	PENG Y, DALMIA S, LANE I, et al. Branchformer: parallel MLP-attention architectures to capture local and global context for speech recognition and understanding[C]//Proceedings of the International Conference on Machine Learning. [S. l.]: PMLR Press, 2022: 17627-17643.
15	李宜亭, 屈丹, 杨绪魁, 等. 基于分解门控注意力单元的高效Conformer模型. 计算机工程, 2023, 49 (5): 73- 80. doi: 10.19678/j.issn.1000-3428.0064687
	LI Y T , QU D , YANG X K , et al. Efficient Conformer model based on factorized gated attention unit. Computer Engineering, 2023, 49 (5): 73- 80. doi: 10.19678/j.issn.1000-3428.0064687
16	YAO Z, GUO L, YANG X, et al. Zipformer: a faster and better encoder for automatic speech recognition[EB/OL]. [2024-06-08]. https://arxiv.org/pdf/2310.11230.
17	胡从刚, 申艺翔, 孙永奇, 等. 基于Conformer的端到端语音识别方法. 计算机应用研究, 2024, 41 (7): 2018- 2024.
	HU C G , SHEN Y X , SUN Y Q , et al. The end-to-end method based on Conformer for speech recognition. Application Research of Computers, 2024, 41 (7): 2018- 2024.
18	KIM Y. Convolutional neural networks for sentence classification[EB/OL]. [2024-06-08]. https://arxiv.org/abs/1408.5882.
19	PARK D S, CHAN W, ZHANG Y, et al. Specaugment: a simple data augmentation method for automatic speech recognition[EB/OL]. [2024-06-08]. https://arxiv.org/abs/1904.08779.
20	LU Y, LI Z, HE D, et al. Understanding and improving Transformer from a multi-particle dynamic system point of view[EB/OL]. [2024-06-08]. https://arxiv.org/abs/1906.02762.
21	WATANABE S, BOYER F, CHANG X K, et al. The 2020 ESPNet update: new features, broadened applications, performance improvements, and future plans[C]//Proceedings of the IEEE Data Science and Learning Workshop (DSLW). Washington D.C., USA: IEEE Press, 2021: 1-6.
22	YAO Z, WU D, WANG X, et al. WeNet: production oriented streaming and non-streaming end-to-end speech recognition toolkit[EB/OL]. [2024-06-08]. https://arxiv.org/abs/2102.01547.
23	HE K M, ZHANG X Y, REN S Q, et al. Deep residual learning for image recognition[C]//Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Washington D.C., USA: IEEE Press, 2016: 770-778.
24	BU H, DU J Y, NA X Y, et al. AISHELL-1: an open-source Mandarin speech corpus and a speech recognition baseline[C]//Proceedings of the 20th Conference of the Oriental Chapter of the International Coordinating Committee on Speech Databases and Speech I/O Systems and Assessment. Washington D.C., USA: IEEE Press, 2017: 1-5.
25	REHR R, GERKMANN T. Cepstral noise subtraction for robust automatic speech recognition[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D.C., USA: IEEE Press, 2015: 375-378.

[1]	庞鑫, 葛凤培, 李艳玲. 声景识音：数字化时代声学场景分类的探索与前沿[J]. 计算机工程, 2025, 51(6): 1-19.
[2]	李宜亭, 屈丹, 杨绪魁, 张昊, 沈小龙. 基于分解门控注意力单元的高效Conformer模型[J]. 计算机工程, 2023, 49(5): 73-80.
[3]	赵德春, 舒洋, 李玲, 陈欢, 张子豪. 丢弃冗余块的语音识别Transformer解码加速方法[J]. 计算机工程, 2023, 49(10): 105-111, 119.
[4]	柏财通, 崔翛龙, 李爱. 基于本地蒸馏联邦学习的鲁棒语音识别技术[J]. 计算机工程, 2022, 48(10): 103-109.
[5]	柏财通, 高志强, 李爱, 崔翛龙. 基于门控网络的军事装备控制指令语音识别研究[J]. 计算机工程, 2021, 47(7): 301-306.
[6]	王恩德,刘巧英,李勇. 基于LLC与GIST特征的静态人体行为分类[J]. 计算机工程, 2018, 44(8): 268-272,278.
[7]	王俊超,黄浩,徐海华,胡英. 基于迁移学习的低资源度维吾尔语语音识别[J]. 计算机工程, 2018, 44(10): 281-285,291.
[8]	胡文君,傅美君,潘文林. 基于Kaldi的普米语语音识别[J]. 计算机工程, 2018, 44(1): 199-205.
[9]	张乐,张雪英,孙颖,张卫. 基于聚合经验模态分解的情感语音特征提取[J]. 计算机工程, 2017, 43(8): 306-309,315.
[10]	项秉伟,景新幸,杨海燕. 基于噪声分类与补偿的车载语音识别[J]. 计算机工程, 2017, 43(3): 220-224.
[11]	商雄伟,张志祥,邱舒婷. 一种通用的限定领域智能语音导学系统设计方法[J]. 计算机工程, 2016, 42(6): 299-304.
[12]	卢良锋,谢志军,叶宏武. 基于RGB特征与深度特征融合的物体识别算法[J]. 计算机工程, 2016, 42(5): 186-193.
[13]	赵彩光,张树群,雷兆宜. 基于改进对比散度的GRBM 语音识别[J]. 计算机工程, 2015, 41(5): 213-218.
[14]	鲜晓东,吕建中,樊宇星. 基于密度与距离参数的CHMM声学模型初值估计[J]. 计算机工程, 2015, 41(10): 318-321.
[15]	张震，赵庆卫，颜永红. 基于语音识别与特征的无监督语音模式提取[J]. 计算机工程, 2014, 40(5): 262-265.

选择文件类型/文献管理软件名称

选择包含的内容

融合最大池化的Conformer中文语音识别

Chinese Speech Recognition Using Conformer Fused with Max Pooling

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 22

参考文献 25

相关文章 15

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

融合最大池化的Conformer中文语音识别

Chinese Speech Recognition Using Conformer Fused with Max Pooling

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 22

参考文献 25

相关文章 15

编辑推荐

Metrics

本文评价