摘要： 由于平行网站的异构性和复杂性,如何自动有效获取双语平行网页以及提高平行网页的质量是构建语料库的关键问题。为此,应用最大熵模型,将平行网页的识别问题看作候选网页对的分类问题,对平行网页的获取方法进行改进。利用基于标题余弦相似性的方法或数据库查询的方法发现候选平行网页对。根据网页内容及候选网页对间余弦相似度特征和最大熵模型训练的分类器对平行网页进行识别。在特征选取上,提取网页的篇章结构特征、词汇化比例特征与页面元素特征等基本特征,并应用TF-IDF算法与余弦相似性提取文档向量的余弦相似度特征。实验结果表明,所提方法可有效提高双语网站中平行网页的召回率和准确率,所获取平行网页的准确率和召回率分别为98%,94%。

关键词: 最大熵模型, 柬埔寨语-英语, 平行网页, 平行语料库, 余弦相似性

Abstract: Because of the isomerism and complexity of parallel pages,how to automatically and effectively get the bilingual parallel pages and how to improve the quality of them are key issues for constructing a parallel corpus.Take example for mining Khmer-English parallel pages,maximum entropy model is implemented to improve parallel pages extraction method.It regards the recognition of parallel pages as the classification of candidate pages.The method is used to find candidate pages based on cosine similarity or database query.The maximum entropy model is trained with the features based on page contents and cosine similarity among candidate pages.Parallel pages are recognized by the classifier.In terms of feature selection,not only the feature of structure,vocabulary and HTML Tag,but also document vector similarity computed by TF-IDF algorithm and cosine similarity is used.Experimental results show this method gains a recall rate of 98% and a precision rate of 94% when collecting parallel pages.

Key words: maximum entropy model, Khmer-English, parallel Web page, parallel corpus, cosine similarity

中图分类号: 

• TP18