基于Squeezeformer的多颗粒度多方面发音质量评测方法

doi:10.19678/j.issn.1000-3428.0068594

摘要/Abstract

摘要：

口语发音质量评测相对于发音错误检测和诊断(MDD)任务, 不仅需要原始的数据特征, 还需要许多流畅度、准确度、完整度等特征辅助进行实现, 所以对口语发音质量评测的研究远远少于对MDD的研究。目前对于口语发音质量评测的研究都是对语音评分某一项指标单方面进行评分。设计将Transformer替换Squeezeformer的改进模型Squeezeformer-MR对基线模型进行改进, Squeezeformer-MR使用多个残差连接增强了前后特征信息的传递。实验中, 在参数设置上保持与基线模型一致, 使用最稳定的24层嵌入层时, 音素级、词级和句子级方面的综合评分的皮尔逊相关系数(PCC)相比基线模型分别提升了1.96%、6.37%和1.08%。在初次改进的基础上, 使用WavLM和HuBERT预训练模型对训练集提取相应的特征, 将提取到的预训练特征以拼接方式添加到原GOP特征中进行特征融合, 使用融合特征以相同方式进行训练, 得到的音素级、词级和句子级方面综合评分的PCC相比基线模型分别提升了2.45%、7.10%和1.89%。

关键词: Squeezeformer模型, 发音质量评测, 预训练模型, 特征融合, 皮尔逊相关系数

Abstract:

Compared with the task of Mispronunciation Detection and Diagnose (MDD), oral pronunciation quality evaluation requires not only the original data features but also additional features such as fluency, accuracy, and completeness. Therefore, research on oral pronunciation quality evaluations is significantly less developed than that on MDD. Current studies on oral pronunciation quality evaluation are based on a single index: the phonetic score. This study replaces the Squeezeformer model with an improved Squeezeformer-MR model based on a Transformer, which enhances the baseline model by exploiting multiple residual connections to improve the transfer of feature information across layers. In the experiments conducted, the parameter settings are consistent with the baseline model. Using the most stable 24-layer embedding layer, the Pearson Correlation Coefficient (PCC) of the comprehensive score increases by 1.96%, 6.37%, and 1.08% at the phoneme, word, and sentence levels, respectively. Building on this improvement, the WavLM and HuBERT pre-training models are employed to extract corresponding features from the training set. These pre-training features are fused with original Goodness of Pronunciation (GOP) features using a splicing method and trained likewise. The fusion features further enhance performance, with PCC improvements of 2.45%, 7.10%, and 1.89% at the phoneme-, word-, and sentence-level scores compared to the baseline model.

Key words: Squeezeformer model, pronunciation quality assessment, pre-training model, feature fusion, Pearson Correlation Coefficient (PCC)

费涛, 艾山·吾买尔, 杜文旭, 朱翠翠. 基于Squeezeformer的多颗粒度多方面发音质量评测方法[J]. 计算机工程, 2025, 51(1): 81-87.

FEI Tao, Aishan Wumaier, DU Wenxu, ZHU Cuicui. Multi-Granularity and Multi-Aspect Pronunciation Quality Evaluation Method Based on Squeezeformer[J]. Computer Engineering, 2025, 51(1): 81-87.

https://www.ecice06.com/CN/Y2025/V51/I1/81

图/表 5

参考文献 25

1	LIN B H, WANG L Y, FENG X L, et al. Automatic scoring at multi-granularity for L2 pronunciation[C]//Proceedings of the Interspeech 2020. [S. l. ]: ISCA, 2020: 3022-3026.
2	LI W, SINISCALCHI S M, CHEN N F, et al. Improving non-native mispronunciation detection and enriching diagnostic feedback with DNN-based speech attribute modeling[C]//Proceedings of the IEEE International Conference on Acoustics, Speech and Signal Processing. Washington D. C., USA: IEEE Press, 2016: 6135-6139.
3	LI K , WU X X , MENG H . Intonation classification for L2 English speech using multi-distribution deep neural networks. Computer Speech [WT《Times New Roman》] & Language, 2017, 43, 18- 33.
4	DOREMALEN J V, CUCCHIARINI C, STRIK H. Using non-native error patterns to improve pronunciation verification[C]//Proceedings of the 11th Annual Conference of the International Speech Communication Association. Makuhari, Japan: [s. n. ], 2010: 26-30.
5	SUDHAKARA S, RAMANATHI M K, YARRA C, et al. An improved Goodness of Pronunciation(GOP) measure for pronunciation evaluation with DNN-HMM system considering HMM transition probabilities[C]//Proceedings of the Interspeech 2019. [S. l. ]: ISCA, 2019: 954-958.
6	HU W P , QIAN Y , SOONG F K , et al. Improved mispronunciation detection with deep neural network trained acoustic models and transfer learning based logistic regression classifiers. Speech Communication, 2015, 67, 154- 166. doi: 10.1016/j.specom.2014.12.008
7	BANNO S, MATASSONI M. Proficiency assessment of L2 spoken English using Wav2Vec 2.0[C]//Proceedings of the IEEE Spoken Language Technology Workshop. Washington D. C., USA: IEEE Press, 2023: 1088-1095.
8	SHI J T, HUO N, JIN Q. Context-aware goodness of pronunciation forcomputer-assisted pronunciation training[EB/OL]. [2023-08-10]. https://arxiv.org/abs/2008.08647 .
9	LIU J J , WUMAIER A , FAN C , et al. Automatic fluency assessment method for spontaneous speech without reference text. Electronics, 2023, 12 (8): 1775. doi: 10.3390/electronics12081775
10	LIU J J, MENG H Z, SHEN Y F, et al. Multimodal automatic speech fluency evaluation method for Putonghua Proficiency Test propositional speaking section[C]//Proceedings of the 13th International Symposium on Chinese Spoken Language Processing. Washington D. C., USA: IEEE Press, 2022: 260-264.
11	WANG Z B , WU Q . RETRACTED: research on automatic evaluation method of Mandarin Chinese pronunciation based on 5G network and FPGA. Microprocessors and Microsystems, 2021, 80, 103534. doi: 10.1016/j.micpro.2020.103534
12	ARIAS J P , YOMA N B , VIVANCO H . Automatic intonation assessment forcomputer aided language learning. Speech Communication, 2010, 52 (3): 254- 267. doi: 10.1016/j.specom.2009.11.001
13	KIM E, JEON J J, SEO H, et al. Automatic pronunciation assessment using self-supervised speech representation learning[C]//Proceedings of the Interspeech 2022. [S. l. ]: ISCA, 2022: 1411-1415.
14	GONG Y, CHEN Z Y, CHU I H, et al. Transformer-based multi-aspect multi-granularity non-native English speaker pronunciation assessment[C]//Proceedings of 2022 IEEE International Conference on Acoustics, Speech and Signal Processing(ICASSP). Washington D. C., USA: IEEE Press, 2022: 7262-7266.
15	DEVLIN J, CHANG M W, LEE K, et al. BERT: pre-training of deep bidirectional transformers for language understanding[EB/OL]. [2023-08-10]. https://arxiv.org/pdf/1810.04805v2 .
16	FAN Z X , LI J , WUMAIER A , et al. A multifaceted approach to oral assessment based on the conformer architecture. IEEE Access, 2023, 11, 28318- 28329. doi: 10.1109/ACCESS.2023.3255986
17	DO H, KIM Y, LEE G G. Hierarchical pronunciation assessment with multi-aspect attention[C]//Proceedings of 2023 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). Washington D. C., USA: IEEE Press, 2023: 1-10.
18	HSU W N , BOLTE B , TSAI Y H H , et al. HuBERT: self-supervised speech representation learning by masked prediction of hidden units. ACM Transactions on Audio, Speech, and Language Processing, 2021, 29, 3451- 3460.
19	CHEN S Y , WANG C Y , CHEN Z Y , et al. WavLM: large-scale self-supervised pre-training for full stack speech processing. IEEE Journal of Selected Topics in Signal Processing, 2022, 16 (6): 1505- 1518. doi: 10.1109/JSTSP.2022.3188113
20	VASWANI A, SHAZEER N, PARMAR N, et al. Attention is all you need[EB/OL]. [2023-08-10]. https://arxiv.org/abs/1706.03762 .
21	WITT S M , YOUNG S J . Phone-level pronunciation scoring and assessment for interactive language learning. Speech Communication, 2000, 30 (2/3): 95- 108.
22	KIM S, GHOLAMI A, SHAW A, et al. Squeezeformer: an efficient transformer for automatic speech recognition[EB/OL]. [2023-08-10]. http://arxiv.org/abs/2206.00888 .
23	ZHANG J B, ZHANG Z W, WANG Y Q, et al. speechocean762: an open-source non-native English speech corpus for pronunciation assessment[EB/OL]. [2023-08-10]. https://arxiv.org/abs/2104.01378 .
24	GULATI A, QIN J, CHIU C C, et al. Conformer: convolution-augmented transformer for speech recognition[EB/OL]. [2023-08-10]. https://arxiv.org/abs/2005.08100
25	BAEVSKI A, ZHOU H, MOHAMED A, et al. wav2vec 2.0: a framework for self-supervised learning of speech representations[EB/OL]. [2023-08-10]. https://arxiv.org/abs/2006.11477 .

[1]	周宇, 谢威, 邝得互, 江健民. 基于三元自注意力的视频快照压缩成像重建[J]. 计算机工程, 2025, 51(1): 20-30.
[2]	王翔, 魏玉锌, 毛国君. 一种融合图数据多元结构和特征的图池化方法[J]. 计算机工程, 2025, 51(1): 128-137.
[3]	胡涌涛, 黄洪琼. 结合特征融合和通道注意力的多分支换装行人重识别[J]. 计算机工程, 2025, 51(1): 225-234.
[4]	李猛坤, 袁晨, 王琪, 赵冲, 陈景轩, 刘立峰. 基于改进YOLOv8算法的在线听课行为识别模型研究[J]. 计算机工程, 2025, 51(1): 287-294.
[5]	魏嵬, 丁香香, 郭梦星, 杨钊, 刘辉. 文本相似度计算方法综述[J]. 计算机工程, 2024, 50(9): 18-32.
[6]	李俊仪, 李向阳, 龙朝勋, 李海燕, 李红松, 余鹏飞. 基于多级区域选择与跨层特征融合的野生菌分类[J]. 计算机工程, 2024, 50(9): 179-188.
[7]	张华青, 夏张涛, 陆晓庆, 童基均. 基于字形特征的血管外科命名实体识别[J]. 计算机工程, 2024, 50(8): 13-21.
[8]	李华昱, 张智康, 闫阳, 岳阳. 基于知识图谱增强的领域多模态实体识别[J]. 计算机工程, 2024, 50(8): 31-39.
[9]	刘锁兰, 王炎, 王洪元, 朱生升. 基于多流语义图卷积网络的人体行为识别[J]. 计算机工程, 2024, 50(8): 64-74.
[10]	赵婉秋, 张俊虎, 李海涛. 用于建筑物分割的平行结构特征融合网络[J]. 计算机工程, 2024, 50(8): 239-248.
[11]	赵宏, 王枭. 基于Swin-Transformer的黑色素瘤图像病灶分割研究[J]. 计算机工程, 2024, 50(8): 249-258.
[12]	王富平, 刘鸿玮, 张锲石, 段冠庄. 基于深度特征抑制的遮挡人脸识别网络[J]. 计算机工程, 2024, 50(8): 259-269.
[13]	闵莉, 董冰洁, 安冬. 基于多注意力机制与跨特征融合的语义分割算法[J]. 计算机工程, 2024, 50(8): 282-289.
[14]	陈宇航, 杨勇, 先木斯亚·买买提明, 帕力旦·吐尔逊, 樊小超, 任鸽, 刁宇峰. 基于主题感知和语义增强的作文自动评分方法[J]. 计算机工程, 2024, 50(8): 363-371.
[15]	谭巨全, 王然. 特征融合下田径录像3D人体动作DTW捕捉算法[J]. 计算机工程, 2024, 50(7): 71-78.

选择文件类型/文献管理软件名称

选择包含的内容