作者投稿和查稿 主编审稿 专家审稿 编委审稿 远程编辑

计算机工程 ›› 2007, Vol. 33 ›› Issue (06): 195-197. doi: 10.3969/j.issn.1000-3428.2007.06.068

• 人工智能及识别技术 • 上一篇    下一篇

信息检索用户查询语句的停用词过滤

熊文新1,2,宋 柔2   

  1. (1. 北京外国语大学中国外语教育研究中心,北京 100089;2. 北京语言大学语言信息处理研究所,北京 100083)
  • 收稿日期:1900-01-01 修回日期:1900-01-01 出版日期:2007-03-20 发布日期:2007-03-20

Removal of Stop Word in Users’ Request for Information Retrieval

XIONG Wenxin1,2, SONG Rou2   

  1. (1. National Research Centre for Foreign Language Education, Beijing Foreign Studies University, Beijing 100089; 2. Center for Language Information Processing, Beijing Language and Culture University, Beijing 100083)
  • Received:1900-01-01 Revised:1900-01-01 Online:2007-03-20 Published:2007-03-20

摘要: 针对以自然语言形式提出的查询请求,区分信息需求表述和信息内容两部分。基于近20万语句的查询语料库和背景语料人民日报对照,提出汉语通用停用词和查询专用的相对停用词,采用左右熵和Ngram方法及KL距离脱机构造相应候选词表。根据候选词语的Bigram属性和句中不同位置的分布特点,给出了在线动态识别停用词的方法。实验结果表明,该文的方法比单纯根据静态停用词表标注效果要好。

关键词: 用户查询, 停用词, 构造, 识别

Abstract: Information need expression and information content words are distinguished for users requests in natural language. Based on the analysis of 200 000 query sentences and the People’s Daily corpus, absolute stop word and relative stop word are proposed. The candidate stop word lists are built offline by means of left/right entropy, Ngram and KL divergence. With the information of Bigram and different position distributions, this paper gives a dynamic identification algorithm for the actual stop word in users’ request expression. The experiment shows the method is superior to the baseline which only consults a stop word list.

Key words: Users request, Stop word, Building, Identification