Abstract: According to the low efficiency of the traditional binary-seek-by-word dictionary mechanism for word segmentation, this paper gives a binary-seek-by-word dictionary mechanism for word segmentation based on all-Hash by analyzing many old dictionary mechanisms. The new mechanism divides the dictionary entry into some groups by character number the entry has, it uses the Hash value of word to reduce the number of string finding. Theoretical analysis and experiment results show that the new mechanism improves the efficiency of word segmentation.

Key words: Chinese segmentation, Hash function, binary-seek-by-word, verbatim binary search, maximum match

摘要： 为提高整词二分词典机制的分词效率，分析现有分词词典机制，提出一种基于全Hash的整词二分词典机制。该机制将首字相同的词条按字数分组，并进行全词Hash，对Hash值相同的词条进行二分查找，从而减少词条匹配的次数。理论分析和实验结果表明，该机制的分词效率较高。

关键词: 中文分词, Hash函数, 整词二分, 逐字二分, 最大匹配

CLC Number: 

• TP391.1