Abstract:
Traditional big data filling algorithms fill missing values depending on the statistical theory of the data set, and they are corrupted by noise data which decrease the imputation accuracy. This paper proposes an algorithm to fill missing values based on distributed incomplete big data clustering. It clusters incomplete big data directly by proposing a new similarity metrics,and uses cloud computing technology to improve clustering efficiency by designing MapReducebased distributed Affinity Propagation(AP) clustering algorithm. The data in the same cluster is utilized to fill missing values. Experimental result demonstrates the proposed algorithm can cluster the incomplete big data directly and improve the filling accuracy of missing data effectively.
Key words:
incomplete big data,
Affinity Propagation ( AP ) clustering,
cloud computing,
data filling,
incomplete
information system
摘要： 传统大数据填充算法是根据整个数据集对缺失数据进行填充,使得填充值容易受到不同类别数据的干扰,
导致填充结果不精确。针对该问题,给出不完整数据的相似度度量方法,使用近邻传播(AP)算法对不完整数据进
行聚类。采用云计算技术优化AP 聚类算法,实现一种基于MapReduce 的分布式聚类算法,根据算法聚类结果将同
一类数据对象划分到相同簇中,并利用同一类对象的属性值对缺失值进行填充。实验结果表明,该算法能实现不
完整大数据的聚类,同时加快聚类速度,提高缺失数据的填充精度。
关键词:
不完整大数据,
近邻传播聚类,
云计算,
数据填充,
不完整信息系统
CLC Number: