计算机工程 ›› 2011, Vol. 37 ›› Issue (15): 174-176.doi: 10.3969/j.issn.1000-3428.2011.15.055

• 人工智能及识别技术 • 上一篇    下一篇

基于Viterbi改进算法的高棉语分词研究

蒋艳荣1,刘习文2,陈耿涛3   

  1. (1. 广东工业大学计算机学院,广州 510006;2. 湘潭大学机械工程学院,湖南 湘潭 411105; 3. 广东国笔科技股份有限公司,广州 510620)
  • 收稿日期:2011-01-10 出版日期:2011-08-05 发布日期:2011-08-05
  • 作者简介:蒋艳荣(1976-),男,讲师、博士,主研方向:文本识别,机器智能;刘习文,副教授、博士;陈耿涛,工程师
  • 基金项目:
    广东省自然科学基金资助项目(8151009001000041)

Research of Khmer Word Segmentation Based on Improved Viterbi Algorithm

JIANG Yan-rong   1, LIU Xi-wen  2, CHEN Geng-tao   3   

  1. (1. Faculty of Computer, Guangdong University of Technology, Guangzhou 510006, China; 2. School of Mechanical Engineering, Xiangtan University, Xiangtan 411105, China; 3. Guangdong Guobi Corporation Ltd., Guangzhou 510620, China)
  • Received:2011-01-10 Online:2011-08-05 Published:2011-08-05

摘要: 采用最大匹配算法对高棉语进行分词准确率较低,且难以正确识别词库中没有的新词。针对该问题,采用改进的Viterbi算法,利用自动机实现音节切分,通过最优选择及剪枝操作提高分词效率,以统计语言模型对未知新词进行数据平滑,提高识别正确率。实验结果表明,改进的Viterbi算法具有较高的分词效率和准确率。

关键词: Viterbi算法, 最大匹配算法, 分词, 高棉语, 剪枝, 统计语言模型

Abstract: The accuracy of Khmer words segmentation for maximum matching algorithm is relatively low, and it is difficult for this algorithm to recognize words that are not enrolled in its dictionary. To solve this problem, an improved Viterbi algorithm is proposed. Wherein automation is used for syllable segmentation, optimization selection and pruning methods are used to promote the segmentation efficiency, and the statistical language model is adopted to perform data smooth for unknown words in this approach. Experimental results indicate that the improved Viterbi algorithm has higher accuracy and efficiency.

Key words: Viterbi algorithm, maximum matching algorithm, word segmentation, Khmer, pruning, statistical language model

中图分类号: