Abstract: Product reviews can help both businesses and consumers make better decisions.The arbitrary nature and irregular grammer structure of user published review makes it difficult for further textual analysis.Aiming at resolving the problem of long sentence segmentation when users omit punctuations,entence segmentation is the foundation of the following text information extraction and text mining work.Since the traditional punctuation-based methods do not work well in this condition,it proposes a machine learning based method to solve this problem.It first extracts candidate segmentation point based on statistical feature of large-scale product review corpus.Then for each candidate segmentation point,its contextual features are extracted as well as the statistical features of product review corpus and employ logistic regression to classify the candidate point.Experimental results show that this method can improve the performance of sentence segmentation when user omits punctuations.

Key words: sentence segmentation, puntuation omitting, machine learning, contextual feature, N-gram, logistic regression

摘要： 商品评论文本对消费者和商家的决策都有重要参考价值。用户在评论中使用的语言较为随意,语法结构不规则,给文本分析带来很大难度。正确的句子切分是文本信息抽取和挖掘工作的基础。为解决商品评论中用户省略标点情况下的句子切分问题,基于上下文特征,提出使用机器学习的方法对评论长句进行切分。根据大规模评论语料的统计特征选取候选句子切分点,对每一个候选句子切分点提取其上下文特征,并根据语料的统计特征,使用逻辑回归对候选切分点进行分类。实验结果表明,该方法能够有效解决商品评论中用户省略标点情况下的句子切分问题。

关键词: 句子切分, 标点省略, 机器学习, 上下文特征, N元文法, 逻辑回归

CLC Number: 

• TP311