摘要： 采用最大匹配算法对高棉语进行分词准确率较低，且难以正确识别词库中没有的新词。针对该问题，采用改进的Viterbi算法，利用自动机实现音节切分，通过最优选择及剪枝操作提高分词效率，以统计语言模型对未知新词进行数据平滑，提高识别正确率。实验结果表明，改进的Viterbi算法具有较高的分词效率和准确率。

关键词: Viterbi算法, 最大匹配算法, 分词, 高棉语, 剪枝, 统计语言模型

Abstract: The accuracy of Khmer words segmentation for maximum matching algorithm is relatively low, and it is difficult for this algorithm to recognize words that are not enrolled in its dictionary. To solve this problem, an improved Viterbi algorithm is proposed. Wherein automation is used for syllable segmentation, optimization selection and pruning methods are used to promote the segmentation efficiency, and the statistical language model is adopted to perform data smooth for unknown words in this approach. Experimental results indicate that the improved Viterbi algorithm has higher accuracy and efficiency.

Key words: Viterbi algorithm, maximum matching algorithm, word segmentation, Khmer, pruning, statistical language model

中图分类号: 

• TP391