作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2011, Vol. 37 ›› Issue (21): 40-42. doi: 10.3969/j.issn.1000-3428.2011.21.014

• 软件技术与数据库 • 上一篇    下一篇

一种基于全Hash的整词二分词典机制

彭焕峰1,丁宋涛2   

  1. (1. 南京工程学院计算机工程学院,南京 211167;2. 南京大学软件学院,南京 210093)
  • 收稿日期:2011-05-16 出版日期:2011-11-05 发布日期:2011-11-05
  • 作者简介:彭焕峰(1978-),男,讲师、硕士,主研方向:大数据量处理,搜索引擎;丁宋涛,讲师、硕士
  • 基金资助:
    南京工程学院科研基金资助项目“基于Lucene的全文搜索引擎研究”(QKJB2009026)

Binary-seek-by-word Dictionary Mechanism Based on All-Hash

PENG Huan-feng 1, DING Song-tao 2   

  1. (1. School of Computer Engineering, Nanjing Institute of Technology, Nanjing 211167, China; 2. Software Institute, Nanjing University, Nanjing 210093, China)
  • Received:2011-05-16 Online:2011-11-05 Published:2011-11-05

摘要: 为提高整词二分词典机制的分词效率,分析现有分词词典机制,提出一种基于全Hash的整词二分词典机制。该机制将首字相同的词条按字数分组,并进行全词Hash,对Hash值相同的词条进行二分查找,从而减少词条匹配的次数。理论分析和实验结果表明,该机制的分词效率较高。

关键词: 中文分词, Hash函数, 整词二分, 逐字二分, 最大匹配

Abstract: According to the low efficiency of the traditional binary-seek-by-word dictionary mechanism for word segmentation, this paper gives a binary-seek-by-word dictionary mechanism for word segmentation based on all-Hash by analyzing many old dictionary mechanisms. The new mechanism divides the dictionary entry into some groups by character number the entry has, it uses the Hash value of word to reduce the number of string finding. Theoretical analysis and experiment results show that the new mechanism improves the efficiency of word segmentation.

Key words: Chinese segmentation, Hash function, binary-seek-by-word, verbatim binary search, maximum match

中图分类号: