摘要： BERT词嵌入模型能够解决简单命名实体识别模型预测精度低的问题，但基于BERT类的复杂词嵌入模型具有计算复杂度高、模型预测时间过长等缺陷。针对该问题，构建基于知识蒸馏的命名实体识别模型，将BERT+CRF模型作为教师模型，获取较高的命名实体识别精度，并基于模型结构相似原则将BiGRU+CRF作为学生模型，在学生模型训练的过程中进行知识蒸馏。知识蒸馏根据教师模型Softmax层和学生模型Softmax层输出的标注概率矩阵分别作为教师模型的知识和学生模型的知识，通过均方损失函数计算教师模型知识与学生模型知识之间的差距，将获得的结果作为软标签误差，将学生模型预测的标签结果与真实标签之间的误差作为硬标签误差，总误差为软标签误差与硬标签误差的加权和，通过误差反向传播进行模型的训练，在减小总误差的同时缩小教师模型知识与学生模型知识之间的差距，使学生模型预测精度接近教师模型。最终使用学生模型进行预测，在接近教师模型预测精度的同时保证相对较短的预测时间。在DuIE2.0数据集上的实验结果表明，该命名实体识别模型在F1值损失2.6%的情况下，可使模型参数规模缩小93.7%，从而缩短了65.2%的运算时间。

关键词: 知识蒸馏, 命名实体识别, 教师模型, 学生模型, BERT模型

Abstract: Currently，to address the low prediction accuracy problem of simple named entity recognition model，the Bidirectional Encoder Representations from Transformers（BERT） word embedding model is widely adopted. However， the complex word embedding model based on BERT suffers from high computational complexity and significant model prediction time. To address this problem，in this study，a named entity recognition model based on knowledge distillation is proposed，which uses a BERT + Conditional Random Field（CRF） model as a teacher model to obtain high named entity recognition accuracy.Using the Bidirectional Gated Recurrent Unit（BiGRU）+CRF as the student model based on the model structure similarity principle.Knowledge distillation uses the annotated probability matrix output taken from the Softmax layer of the teacher model as the knowledge of the teacher model and that taken from the Softmax layer of the student model as the knowledge of the student model. Subsequently，the gap between the knowledge of the teacher model and that of the student model is calculated using the mean-square loss function as soft-label errors.The error between the label results predicted by the student model and real label is denoted as the hard label error. The total error is the weighted combination of the soft label and hard label errors. Finally，the student model is trained through error backpropagation， reducing the difference between the knowledge of the teacher model and that of the student model while narrowing the total error. This ensures that the accuracy of the student model prediction is close to that of the teacher model. Finally，the student model is used for prediction，which ensures a relatively short prediction time and high prediction accuracy.The experimental results demonstrate that，using the dataset DuIE2.0，the scale of model parameters is reduced by 93.7% and operation time is reduced by 65.2% under the condition of losing 2.6% of F1 value.

Key words: knowledge distillation, named entity recognition, teacher model, student model, BERT model

中图分类号: 

• TP18