摘要： Hadoop平台目前有近200个配置参数,对这些参数进行合理配置能提高系统性能。针对Hadoop参数配置的优化问题,提出一种基于递归随机抽样的黑盒优化策略。利用随机抽样的初始高效性,通过不断调整样本空间进行递归随机抽样,从而快速搜索到近似的全局最优配置。实 验结果表明,与传统配置方法相比,应用黑盒优化策略的配置方法可提高14%~25%的Hadoop作业处理速度,且具有较好的稳定性和可靠性。

关键词: Hadoop平台, 黑盒优化, 粒子群算法, 模拟退火算法, 递归随机抽样

Abstract: Hadoop platform has over 200 configuration parameters,it can improve the performance of the system by deploying these parameters in reason.Aiming at the optimization of Hadoop configuration parameters problem,a black box optimization strategy based on recursive random sampling is proposed.It utilizes the initial high efficiency of random sampling,constantly adjusts the sample space to start recursive random sampling,and searches the approximate global optimal configuration quickly and efficiently.Experimental results show that,the configuration method based on black box optimization strategy can improve the job processing speed by 14%~25% than traditional method,and it has good stability and reliability.

Key words: Hadoop platform, black box optimization, article swarm algorithm, simulated annealing algorithm, recursive random sampling

中图分类号: 

• TP302