摘要： 新词/短语的识别是自然语言处理、信息检索和机器翻译等领域的一项基础研究。该文分析了已有短语抽取技术,并结合汉语特点,提出了基于概率统计技术和规则方法相结合的概念抽取方法。该方法包括高效的“二元语法”统计模型、统计算法、统计选词策略、丰富的规则知识和规则过滤算法。实验证明该方法适用于从大规模语料库中自动高效地发现新词/短语。

关键词: 新词发现, 短语抽取, 二元语法, 语料库

Abstract: New word and phrase discovery is basic research in fields of NLP, IR and MT. The paper gives the method based on probabilistic techniques and rules for new word discovery via analyzing the current techniques of phrase extraction and combining the specialties of Chinese. This method includes the "bi-gram" probabilistic model, the statistical algorithm, the rich rules and rule-based algorithm for word filtering. Experiments show that this technique is fit for automatically and effectually extracting new words/phrases from large corpora.

中图分类号: 

• TP391.4