基于Spark平台的ALS加速算法研究

doi:10.19678/j.issn.1000-3428.0054147

计算机工程 ›› 2020, Vol. 46 ›› Issue (2): 103-109. doi: 10.19678/j.issn.1000-3428.0054147

基于Spark平台的ALS加速算法研究

贾晓芳, 桑国明, 祁文凯

大连海事大学信息科学技术学院, 辽宁大连 116026

收稿日期:2019-03-08 修回日期:2019-05-06 发布日期:2019-06-06
作者简介:贾晓芳(1994-),女,硕士研究生,主研方向为大数据应用、数据挖掘算法;桑国明(通信作者),副教授;祁文凯,硕士研究生。
基金资助:
国家自然科学基金（61672122）；中央高校基本科研业务费项目"大规模协作式多智能体强化学习技术研究"（3132019207）。

Research on ALS Acceleration Algorithm Based on Spark Platform

JIA Xiaofang, SANG Guoming, QI Wenkai

School of Information Science and Technology, Dalian Maritime University, Dalian, Liaoning 116026, China

Received:2019-03-08 Revised:2019-05-06 Published:2019-06-06

摘要/Abstract

摘要： 协同过滤推荐算法在推荐系统中发挥着重要作用，但其存在执行效率与排名精度较低的问题，交替最小二乘（ALS）算法可实现并行计算，从而提高执行效率，但是该算法数据加载与迭代收敛的时间较长。为此，将非线性共轭梯度（NCG）算法与ALS算法相结合，提出一种ALS-NCG算法，以达到加速ALS算法的目的。在Spark分布式数据处理环境中对ALS-NCG算法进行性能评估，实验结果表明，相比ALS算法，ALS-NCG算法获取高精度推荐排名时需要的迭代次数与时间更少。

Abstract: Collaborative filtering algorithm plays an important role in recommendation system,but its execution efficiency and ranking accuracy are both low.Alternating Least Squares(ALS) algorithm can implement parallel computing,thus improving the execution efficiency,but the time between data loading and iterative convergence of the algorithm is a bit long.Therefore,by combing the Nonlinear Conjugate Gradient(NCG) algorithm and the ALS algorithm,this paper proposes an ALS-NCG algorithm to accelerate the ALS algorithm.The performance of the ALS-NCG algorithm is evaluated in the Spark distributed data processing environment.Experimental results show that compared with the ALS algorithm,the ALS-NCG algorithm needs less iterations and time to obtain high-precision recommended ranking.

Key words: collaborative filtering, recommendation algorithm, Alternating Least Squares(ALS) algorithm, Nonlinear Conjugate Gradient(NCG), Spark platform

中图分类号:

TP312

贾晓芳, 桑国明, 祁文凯. 基于Spark平台的ALS加速算法研究[J]. 计算机工程, 2020, 46(2): 103-109.

JIA Xiaofang, SANG Guoming, QI Wenkai. Research on ALS Acceleration Algorithm Based on Spark Platform[J]. Computer Engineering, 2020, 46(2): 103-109.

https://www.ecice06.com/CN/Y2020/V46/I2/103

图/表 12

20200218141937

20200218141941

20200218141944

20200218141947

20200218141951

20200218141954

20200218141958

20200218142000

20200218142004

20200218142007

20200218142010

20200218142014

参考文献

[1] GANTZ J,REINSEL D.The digital universe in 2020:big data,bigger digital shadows,and biggest growth in the far east[EB/OL].[2019-02-10].https://www.speicherguide.de/download/dokus/IDC-Digital-Universe-Studie-iView-11.12.pdf.
[2] ISINKAYE F O,FOLAJIMI Y O,OJOKOH B A.Recommendation systems:principles,methods and evaluation[J].Egyptian Informatics Journal,2015,16(3):261-273.
[3] KOREN Y,BELL R,VOLINSKY C.Matrix factorization techniques for recommender systems[J].Computer,2009,42(8):30-37.
[4] LIU Yingnan,XIE Jinkui,ZHANG Jiali,et al.Recommendation algorithm based on trust in social network[J].Journal of Chinese Computer Systems,2015,36(6):1165-1170.(in Chinese) 刘英南,谢瑾奎,张家利,等.社交网络中基于信任的推荐算法[J].小型微型计算机系统,2015,36(6):1165-1170.
[5] BOKDE D,GIRASE S,MUKHOPADHYAY D.Matrix factorization model in collaborative filtering algorithms:a survey[J].Procedia Computer Science,2015,49(1):136-146.
[6] MENG Xiangwu,LIANG Bi,DU Yulu,et al.Study on the utility evaluation of location-based mobile recom-mendation system[J].Chinese Journal of Computers,2019,42(12):2695-2721.(in Chinese) 孟祥武,梁弼,杜雨露,等.基于位置的移动推荐系统效用评价研究[J].计算机学报,2019,42(12):2695-2721.
[7] HERNANDO A,BOBADILLA J,ORTEGA F.A non negative matrix factorization for collaborative filtering recommender systems based on a Bayesian probabilistic model[J].Knowledge-Based Systems,2016,97(4):188-202.
[8] DU Dongfang,XU Tong,LU Yanan,et al.User rating prediction based on trust-driven probabilistic matrix factorization[J].Journal of Software,2018,29(12):3747-3763.(in Chinese) 杜东舫,徐童,鲁亚男,等.基于信任机制下概率矩阵分解的用户评分预测[J].软件学报,2018,29(12):3747-3763.
[9] KOREN Y.Collaborative filtering with temporal dynamics[J].Communications of the ACM,2010,53(4):447-456.
[10] DESHPANDE M,GEORGE K.Item-based top-n recommendation algorithms[J].ACM Transactions on Information Systems,2004,22(1):143-177.
[11] WANG Jianfang,MIAO Yanling,HAN Pengfei,et al.Probabilistic matrix factorization algorithm of colla-borative filtering based on trust mechanism[J].Journal of Chinese Computer Systems,2019,40(1):31-35.(in Chinese) 王建芳,苗艳玲,韩鹏飞,等.一种基于信任机制的概率矩阵分解协同过滤推荐算法[J].小型微型计算机系统,2019,40(1):31-35.
[12] CONDIE T,MINEIRO P,POLYZOTIS N,et al.Machine learning on big data[C]//Proceedings of 2013 IEEE International Conference on Data Engineering.Washington D.C.,USA:IEEE Press,2013:1242-1244.
[13] TU Xiaohan,LIU Siping,LI Renfa.Improving matrix factorization recommendations for problems in big data[C]//Proceedings of 2017 IEEE International Conference on Big Data Analysis.Washington D.C.,USA:IEEE Press,2017:193-197.
[14] THAN H A,RACHSUDA J T.Alternating least squares with incremental learning bias[C]//Proceedings of 2015 International Joint Conference on Computer Science and Software Engineering.Washington D.C.,USA:IEEE Press,2015:297-302.
[15] STERCK H D,WINLAW M.A nonlinearly preconditioned conjugate gradient algorithm for rank-R canonical tensor approximation[J].Numerical Linear Algebra with Applications,2015,22(3):410-432.
[16] YE Hanmin,ZHANG Qiuling,BAI Xue.A new collaborative filtering algorithm based on modified matrix factorization[C]//Proceedings of 2017 IEEE Advanced Information Technology Electronic and Automation Control Conference.Washington D.C.,USA:IEEE Press,2017:147-151.
[17] ZHOU Y H,WILKINSON D,SCHREIBER R,et al.Large-scale parallel collaborative filtering for the netflix prize[C]//Proceedings of the 4th International Conference on Algorthmic Aspects in Information and Management.Berlin,Germany:Springer,2008:337-348.
[18] QI Changxia.Study on several fusion nonlinear conjugate gradient methods[D].Qinhuangdao:Yanshan University,2017.(in Chinese) 齐昌霞.几种融合非线性共轭梯度法的研究[D].秦皇岛:燕山大学,2017.
[19] LIU Zhengrong.Several new nonlinear conjugate gradient methods[D].Nanning:Guangxi University,2016.(in Chinese) 刘峥嵘.几种新的非线性共轭梯度法[D].南宁:广西大学,2016.
[20] ZAHARIA M,CHOWDHURY M,DAS T,et al.Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computerin[C]//Proceedings of USENIX Conference on Networked Systems Design and Implementation.San Diego,USA:USENIX Association,2012:141-146.
[21] WU Xindong,JI Shengwei.Comparative study on MapReduce and Spark for big data analytics[J].Journal of Software,2018,29(6):1770-1791.(in Chinese) 吴信东,嵇圣硙.MapReduce与Spark用于大数据分析之比较[J].软件学报,2018,29(6):1770-1791.

选择文件类型/文献管理软件名称

选择包含的内容

基于Spark平台的ALS加速算法研究

Research on ALS Acceleration Algorithm Based on Spark Platform

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	王明虎, 石智奎, 苏佳, 张新生. 基于RoBERTa和图增强Transformer的序列推荐方法[J]. 计算机工程, 2024, 50(4): 121-131.
[2]	顾嘉静, 杨丹, 聂铁铮, 寇月. 基于多视图融合跨层对比学习的推荐算法[J]. 计算机工程, 2024, 50(1): 120-128.
[3]	许凤, 杨兴耀, 于炯, 李梓杨, 李晨瑜, 张君. 小波卷积增强的对比学习推荐算法[J]. 计算机工程, 2023, 49(5): 105-111,121.
[4]	徐上上, 孙福振, 王绍卿, 董家玮, 吴田慧. 基于图神经网络的异构信任推荐算法[J]. 计算机工程, 2022, 48(9): 89-95,104.
[5]	苗雨欣, 宋春花, 牛保宁, 康瑞雪. 双通道图协同过滤推荐算法[J]. 计算机工程, 2022, 48(8): 121-128.
[6]	王燕, 范林, 赵妮妮. 利用门控网络构建用户动态兴趣的序列推荐模型[J]. 计算机工程, 2022, 48(8): 283-291.
[7]	胡承佐, 王庆梅, 李迪超, 王铮. 基于复杂结构信息的图神经网络序列推荐算法[J]. 计算机工程, 2022, 48(5): 82-90,97.
[8]	沈学利, 马玉营, 梁振兴. 融合复杂先验与注意力机制的变分自动编码器[J]. 计算机工程, 2022, 48(11): 55-61.
[9]	宋玉龙, 马文明, 刘彤彤. 融合用户信任度的概率矩阵分解群组推荐算法[J]. 计算机工程, 2022, 48(1): 105-111.
[10]	刘峰, 王宝亮, 邹荣宇, 赵浩淳. 基于随机游走的网络表示学习推荐算法[J]. 计算机工程, 2021, 47(9): 90-96,105.
[11]	贾俊杰, 刘鹏涛, 陈旺虎. 融合社交信息的矩阵分解改进推荐算法[J]. 计算机工程, 2021, 47(9): 97-105.
[12]	刘峰, 王宝亮, 潘文采. 基于网络表示学习与深度学习的推荐算法研究[J]. 计算机工程, 2021, 47(8): 54-61.
[13]	李昆仑, 于志波, 翟利娜, 赵佳耀. 基于注意力机制与改进TF-IDF的推荐算法[J]. 计算机工程, 2021, 47(8): 69-77.
[14]	陈恩华, 方宝富. 基于物品嵌入向量的会话型推荐算法[J]. 计算机工程, 2021, 47(7): 74-80.
[15]	张浩博, 薛峰, 刘凯. 基于半自动编码器的协同过滤推荐算法[J]. 计算机工程, 2021, 47(3): 125-130.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于Spark平台的ALS加速算法研究

Research on ALS Acceleration Algorithm Based on Spark Platform

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 12

参考文献

相关文章 15

编辑推荐

Metrics

本文评价