基于网站层次结构和主题模型LDA的网站自动摘要

doi:10.3969/j.issn.1000-3428.2017.04.035

计算机工程

基于网站层次结构和主题模型LDA的网站自动摘要

李舒嫒,杨静,顾君忠

(华东师范大学计算机科学技术系,上海 200241)

收稿日期:2016-04-18 出版日期:2017-04-15 发布日期:2017-04-14
作者简介:李舒嫒(1993-),女,硕士研究生,主研方向为数据挖掘;杨静,副教授;顾君忠,教授。
基金资助:
国家科技支撑计划项目(2015BAH01F02);上海张江国家自主创新示范区专项发展资金计划项目(201411-JA-B108-002)。

Website Automatic Summarization Based on Website Hierarchy and Latent Dirichlet Allocation

LI Shu’ai,YANG Jing,GU Junzhong

(Department of Computer Science and Technology,East China Normal University,Shanghai 200241,China)

Received:2016-04-18 Online:2017-04-15 Published:2017-04-14

摘要/Abstract

摘要：

近年来自动摘要方面的研究大多是关于多文档和Web网页的,而对网站自动摘要的研究较少。为此,基于主题模型隐含狄利克雷分布(LDA)和网站层次结构提出一个可以自动生成网站摘要的算法。该算法可获取整个网站内的网页信息并进行整合,根据提出的句子权重公式计算句子权重,选取权重最高的句子作为网站摘要。以20个商业和学术网站作为实验对象,使用ROUGE评测标准,结果表明,与仅使用主题模型LDA获取的网站摘要相比,不带停用词的ROUGE-1和ROUGE-L提高 0.32,带停用词的ROUGE-1提高0.39,ROUGE-L提高0.38。与网站首页摘要相比,不带停用词的ROUGE-1提高 0.03,ROUGE-L提高0.06,带停用词的ROUGE-1提高 0.08,ROUGE-L提高0.07。

关键词: Web网页, 网站自动摘要, 隐含狄利克雷分布, 网站层次结构, 宽度优先搜索

Abstract:

In recent years,the research of automatic summarization is mostly about multi-documents and Web pages,but less about website summarization.A method that summarizes a website automatically based on the hierarchical structure of the website and Latent Dirichlet Allocation is proposed.This method gets the information from web pages in the given website and fuses it,and calculates the weight of sentences according to the proposed sentence weighting formula,and selects the highest weight sentences as the website summarization.An experiment is done based on 20 commercial websites and academic websites,and using ROUGE evaluation.Results show that compared with the summaries only using LDA,ROUGE-1 and ROUGE-L are increased by 0.32 with no stop words;ROUGE-1 is increased by 0.39 and ROUGE-L is increased by 0.38 with stop words.Compared with the summaries only from homepage,ROUGE-1 is increased by 0.03 and ROUGE-L is increased by 0.06 with no stop words;ROUGE-1 is increased by 0.08 and ROUGE-L is increased by 0.07 with stop words.

Key words: Web pages, website automatic summarization, Latent Dirichlet Allocation(LDA), website hierarchy, breadth-first search

中图分类号:

TP18

李舒嫒,杨静,顾君忠. 基于网站层次结构和主题模型LDA的网站自动摘要[J]. 计算机工程, doi: 10.3969/j.issn.1000-3428.2017.04.035.

LI Shu’ai,YANG Jing,GU Junzhong. Website Automatic Summarization Based on Website Hierarchy and Latent Dirichlet Allocation[J]. Computer Engineering, doi: 10.3969/j.issn.1000-3428.2017.04.035.

http://www.ecice06.com/CN/Y2017/V43/I4/207

参考文献

参考文献［1］郭磊,马军,陈竹敏,等.一种结合推荐对象间关联关系的社会化推荐算法［J］.计算机学报,2014,37(1):219-228. ［2］邹本友,李翠平,谭力文,等.基于用户信任和张量分解的社会网络推荐［J］.软件学报,2014,25(12):2852-2864. ［3］Cantador I,Bellogín A,Vallet D.Content-based Recommendation in Social Tagging Systems［C］//Proceedings of the 4th ACM Conference on Recom-mender Systems.New York,USA:ACM Press,2010:237-240. ［4］Rafailidis D,Daras P.The TFC Model:Tensor Factorization and Tag Clustering for Item Recom-mendation in Social Tagging Systems［J］.IEEE Transac-tions on Systems,Man,and Cybernetics Systems,2013,43(3):673-688. ［5］Liu Rujuan,Niu Zhendong.A Collaborative Filtering Recommendation Algorithm Based on Tag Cluster-ing［M］//Park J J,Stojmenovic I,Choi M,et al.Future Information Technology.Berlin,Germany:Springer,2014:177-183. ［6］马费成,张斌.图书标注环境下用户的认知特征［J］.中国图书馆学报,2014(1):4-14. ［7］王军,张子柯.基于社会化标签信息熵的个性化推荐算法［J］.图书情报工作,2013(23):31-35. ［8］琚春华,鲍福光,刘中军.基于社会化评分和标签的个性化推荐方法［J］.情报学报,2014(12):1302-1310. ［9］Durao F,Dolog P.A Personalized Tag-based Recom-mendation in Social Web Systems［C］//Proceedings of International Workshop on Adaptation & Personalization for Web.Washington D.C.,USA:IEEE Press,2012:40-49. ［10］孔欣欣,苏本昌,王宏志,等.基于标签权重评分的推荐模型及算法研究［J］.计算机学报,2015,38(23):1-13. ［11］房小可,纪春光.基于标签主题和概念空间的个性化推荐研究［J］.情报理论与实践,2015,38(5):105-111. ［12］Zhang Zike,Zhou Tao,Zhang Yicheng.Personalized Recommendation via Integrated Diffusion on User-Item-Tag Tripartite Graphs［J］.Statistical Mechanics and Its Applications,2010,389(1):179-186. ［13］Zhang Yin,Zhang Bin,Gao Kening,et al.Combining Content and Relation Analysis for Recommendation in Social Tagging Systems［J］.Statistical Mechanics and Its Applications,2012,391(22):5759-5768. ［14］Lian Jie,Liu Yun,Zhang Zhenjiang,et al.Personalized Recommendation via an Improved NBI Algorithm and User Influence Model in a Microblog Network［J］.Statistical Mechanics and Its Applications,2013,392(19):4594-4605. ［15］Zhang Zike,Yu Lu,Fang Kuan,et al.Website-oriented Recommendation Based on Heat Spreading and Tag-aware Collaborative Filtering［J］.Statistical Mechanics and Its Applications,2014,393(4):82-88. ［16］Mao Jin,Lu Kun,Li Gang,et al.Profiling Users with Tag Networks in Diffusion-based Personalized Recom-mendation［J］.Journal of Information Science,2016,42(5):75-89. 编辑顾逸斐

[1]	李琴, 李少波, 胡杰. 基于主题情感联合分析的游客画像研究[J]. 计算机工程, 2022, 48(6): 278-287,294.
[2]	张聪, 易秀双, 朱明浩, 王兴伟. 基于Spark的学术研究热点挖掘方法[J]. 计算机工程, 2019, 45(12): 171-175.
[3]	肖宝,李璞,胡娇娇,蒋运承. 基于潜在语义与图结构的微博语义检索[J]. 计算机工程, 2017, 43(6): 182-188,194.
[4]	孙玉洁,秦永彬. 基于LDA模型的多角度个性化微博推荐算法[J]. 计算机工程, 2017, 43(4): 177-182.

选择文件类型/文献管理软件名称

选择包含的内容

基于网站层次结构和主题模型LDA的网站自动摘要

Website Automatic Summarization Based on Website Hierarchy and Latent Dirichlet Allocation

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

编辑推荐

Metrics

本文评价

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于网站层次结构和主题模型LDA的网站自动摘要

Website Automatic Summarization Based on Website Hierarchy and Latent Dirichlet Allocation

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 4

编辑推荐

Metrics

本文评价