计算机工程 ›› 2013, Vol. 39 ›› Issue (7): 73-75,82.doi: 10.3969/j.issn.1000-3428.2013.07.016

• 先进计算与数据处理 • 上一篇    下一篇

基于声学指纹的海量MP3文件近似去重方法

赵晓永,杨 扬,王 宁   

  1. (北京科技大学计算机与通信工程学院,北京 100083)
  • 收稿日期:2012-07-12 出版日期:2013-07-15 发布日期:2013-07-12
  • 作者简介:赵晓永(1981-),男,博士研究生、CCF会员,主研方向:云存储;杨 扬,教授、博士生导师;王 宁,博士研究生
  • 基金项目:
    国家自然科学基金资助项目(61070182, 61170209)

Near De-duplication Method of Massive MP3 Files Based on Acoustic Fingerprint

ZHAO Xiao-yong, YANG Yang, WANG Ning   

  1. (School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing 100083, China)
  • Received:2012-07-12 Online:2013-07-15 Published:2013-07-12

摘要: 在互联网中重复上传他人已经分享的歌曲会消耗网络带宽,浪费存储空间,但目前的重复数据删除方法主要基于文件的二进制特征,无法识别经过信号处理或压缩后的歌曲。针对该问题,提出一种基于声学指纹的海量MP3文件近似去重方法。结合文件消息摘要的确定性与声学指纹的鲁棒性,在采用布隆过滤器对文件消息摘要一次去重的基础上,根据降维后的声学指纹值进行二次近似去重,保证高效的同时提高去重率。实验结果表明,与可变分块检测方法相比,该方法的去重率可提高1倍以上,扩展性较好。

关键词: 声学指纹, 重复数据删除, 近似去重, 布隆过滤器, 海量数据

Abstract: Song re-uploading had been shared wastes network bandwidth and storage, which needs to use data de-duplication technology. However, the current approach to de-duplication based file bit-feature does not recognize the same song after signal processing or compression. Aiming at this problem, this paper proposes a near de-duplication method of massive MP3 files based on acoustic fingerprint. It combines the certainty of message digest with the robustness of the acoustic fingerprint, after Bloom Filter(BF) de-duplicate data based on the message digest, then reduces acoustic fingerprint for the secondary near de-duplication based on the dimensionality. It ensures efficient at the same time, greatly improves the de-duplication ratio. Experimental results show that this method can improve the de-duplication rate by one time than Content-defined Chunking(CDC) method, and has good extensibility.

Key words: acoustic fingerprint, data de-duplication, near de-duplication, Bloom Filter(BF), massive data

中图分类号: