摘要： 针对来源于多个业务系统的重复主数据影响主数据质量、主数据同步及主数据挖掘等问题，提出重复主数据检测算法fastCdrDetection。从数据可信度的角度出发，在考虑数据源可信度、数据最后更新时间及数据长度的基础上，建立主数据可信度模型，并实现可信记录生成算法。设计非递归的字符串相似度计算算法FiledMatch，解决了由中文简写、缩写、错误拼写造成的主数据重复问题，采用sourceKeys算法对来源于同一业务系统、具有同样业务主键的重复记录进行预处理，从而提高重复主数 据检测效率。通过对某电网基建物资63万余条供应商存量数据及23万余条模拟数据进行实验，结果表明，与PQS算法相比，fastCdrDetection算法的召回率由74%提高到88%，准确率由61%提高到95%，证明了算法的有效性。
关键词:
多数据源,
重复主数据,
可信度模型,
检测算法,
数据可信度
Abstract: To avoid the effect of duplicate master data from multiple business systems on the quality, synchronization of the master data as well as master data mining, this paper propose a fastCdrDetection(Fast Cluster Duplicate Records Detection) algorithm, in which a duplicate master data detection model and a credible record generating algorithm are included, considering data source reliability, data refresh time and data length. A non-recursive algorithm FiledMatch is established for character string similarity calculation. Aiming at the eliminating problems caused by abbreviations and wrong spellings in Chinese input, a sourceKeys algorithm is constructed for pretreatment of duplicate records arising from a same business system and sharing same business keys to achieve high efficiency in duplicate master data detection. Experiments are carried on a power grid with 630 thousand records of raw material and 230 thousand simulated data records. Result shows that the recall rate of the fastCdrDetection algorithm is 88%, while the PQS algorithm is 74%, and the accuracy is 95% to 61%. The effectiveness of the algorithm is verified.
Key words:
multiple data source,
duplicate master data,
credibility model,
detection algorithm,
data credibility
中图分类号: