摘要： 压缩模糊K-近邻（CFKNN）算法仅适用于中小数据环境，且其样例选择采用静态机制，导致算法不能对阈值进行动态调整从而选出最优样例。为此，对CFKNN算法进行改进，将其扩展到大规模数据环境，提出分别基于MapReduce和Spark的2种大规模压缩模糊K-近邻算法。在样例选择阈值设置方面，引入动态机制，使得所选样例更具代表性。在具有7个数据节点的大数据平台上进行实验，结果表明，与CFKNN算法相比，所提2种算法具有更高的分类精度和加速比。2个平台相比，MapReduce产生的中间文件数目多于Spark，而Spark在运行时间和同步次数上优于MapReduce。

关键词: MapReduce平台, Spark平台, 模糊K-近邻, 样例选择, 动态机制

Abstract: The Condensed Fuzzy K-Nearest Neighbor(CFKNN) algorithm is applicable only to small-sized and medium-sized data sets,and its mechanism of instance selection is static,so the algorithm can not adjust the threshold dynamically to select the optimal sample.To address the problems,this paper improves the CFKNN algorithm to make it applicable it to large-scale data environment,and on this basis proposes two kinds of large-scale condensed fuzzy K-nearest neighbor algorithms based on MapReduce and Spark.A dynamic mechanism is introduced into the setting of the instance selection threshold to make the selected instances more representative.Experiments are carried out on a big data platform with 7 data nodes.The experimental results show that compared with the CFKNN algorithm,the two proposed algorithms have better classification accuracy and acceleration ratio.Results of comparison between the two proposed algorithms show that MapReduce produces more intermediate files than Spark,yet Spark outperforms MapReduce in terms of running time and synchronization times.

Key words: MapReduce platform, Spark platform, fuzzy K-Nearest Neighbor(K-NN), instance selection, dynamic mechanism

中图分类号: 

• TP311