基于Petri网的数据清洗规则链自动组合与检测

doi:10.19678/j.issn.1000-3428.0056429

计算机工程 ›› 2020, Vol. 46 ›› Issue (11): 124-131. doi: 10.19678/j.issn.1000-3428.0056429

基于Petri网的数据清洗规则链自动组合与检测

何俊¹, 张云飞¹, 张德海²

1. 昆明学院信息工程学院, 昆明 650214;
2. 云南大学软件学院, 昆明 650206

收稿日期:2019-10-28 修回日期:2019-12-01 发布日期:2019-12-09
作者简介:何俊(1977-),男,副教授、博士,主研方向为数据分析;张云飞,讲师、硕士;张德海,副教授、博士。
基金资助:
国家自然科学基金（61263043，61864004）；云南省地方本科高校基础研究联合专项（2017FH001-05）。

Automatic Combination and Detection of Data Cleaning Rule Chains Based on Petri Net

HE Jun¹, ZHANG Yunfei¹, ZHANG Dehai²

1. College of Information Engineering, Kunming University, Kunming 650214, China;
2. College of Software, Yunnan University, Kunming 650206, China

Received:2019-10-28 Revised:2019-12-01 Published:2019-12-09

摘要/Abstract

摘要： 针对传统规则链顺序执行方法面向大规模数据清洗任务时存在的规则冗余和逻辑冲突问题，提出一种规则链自动组合与检测方法。结合上下文信息设计通用、领域和自定义的三层规则库，基于Petri网建立规则链组合模型（RCCM），实现规则链自动生成、逻辑正确性与状态可达性检测以及规则链优选。以某地区扶贫领域的数据清洗应用为例，通过RCCM模型实现的实验结果表明，该方法能明显减少错误数据的产生，提高数据清洗质量和效率。

关键词: 数据清洗, 分层规则库, 规则链, Petri网, 逻辑冲突

Abstract: To address the rule redundancy and logical conflicts of the sequential execution method of traditional rule chains applied to massive Data Cleaning(DC) task,this paper proposes an automatic combination and detection method for rule chains.A three-layer rule base including general,field-specific and customized layers is designed based on the context information.Then a Rule Chain Combination Model(RCCM) is established based on Petri Net(PN) to realize the automatic generation of rule chains,the detection of logical correctness and state accessibility,as well as the optimization of rule chains.The proposed method takes the DC application in the field of poverty alleviation in a certain area as an example.Experimental results on RCCM implementation show that the proposed method can significantly reduce generated error data and improves the quality and efficiency of DC.

Key words: Data Cleaning(DC), hierarchical rule base, rule chain, Petri Net(PN), logical conflict

中图分类号:

TP391.1

何俊, 张云飞, 张德海. 基于Petri网的数据清洗规则链自动组合与检测[J]. 计算机工程, 2020, 46(11): 124-131.

HE Jun, ZHANG Yunfei, ZHANG Dehai. Automatic Combination and Detection of Data Cleaning Rule Chains Based on Petri Net[J]. Computer Engineering, 2020, 46(11): 124-131.

http://www.ecice06.com/CN/Y2020/V46/I11/124

图/表 9

20201124085410

20201124085413

20201124085416

20201124085419

20201124085423

20201124085425

20201124085428

20201124085431

20201124085435

参考文献

[1] WU Xindong,DONG Bingbing,DU Xinzheng,et al.Data governance technology[J].Journal of Software,2019,30(9):2830-2856.(in Chinese)吴信东,董丙冰,堵新政,等.数据治理技术[J].软件学报,2019,30(9):2830-2856.
[2] ZHU Huijuan,JIANG Tonghai,ZHOU Xi,et al.Data cleaning method based on dynamic configurable rules[J].Computer Application,2017,37(4):1014-1020.(in Chinese)朱会娟,蒋同海,周喜,等.基于动态可配置规则的数据清洗方法[J].计算机应用,2017,37(4):1014-1020.
[3] RAHM E,DO H.Data cleaning:problems and current approaches[J].IEEE Data Engineering Bulletin,2000,23(4):3-13.
[4] TANG N.Big data cleaning[M]//SHENG Q Z,WANG G R,JENSEN C S.Web technologies and applications.Berlin,Germany:Springer,2014:13-24.
[5] LEE M L,LING T W,LOW W L.IntelliClean:a knowledge based intelligent data cleaner[C]//Proceedings of the 6th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining.New York,USA:ACM Press,2000:290-294.
[6] MONGE A E.Matching algorithms within a duplicate detection system[J].IEEE Data Engineering Bulletin,2000,23(4):14-20.
[7] PAPOTTI P,CHU X,ILYAS I F.Holistic data cleaning:putting violations into context[C]//Proceedings of the 29th International Conference on Data Engineering.Washington D.C.,USA:IEEE Press,2013:458-469.
[8] CAO Jianjun,DIAO Xingchun,WANG Ting,et al.Research on domain-independent data cleaning:a survey[J].Computer Science,2010,37(5):26-29.(in Chinese)曹建军,刁兴春,汪挺,等.领域无关数据清洗研究综述[J].计算机科学,2010,37(5):26-29.
[9] SALEM R,ABDO A.Fixing rules for data cleaning based on conditional functional dependency[J].Future Computing and Informatics Journal,2016,1(1/2):10-26.
[10] LEDERER C,ALTSTADT S,ANDRIAMONJE S.Web usage data cleaning:a rule-based approach for Weblog data cleaning[C]//Proceedings of the 20th International Conference on Big Data Analytics and Knowledge Discovery.Washington D.C.,USA:IEEE Press,2018:1-10.
[11] YANG Xue,TANG Luliang,ZHANG Xia,et al.A data cleaning method for big trace data using movement consistency[J].Sensors,2018,18(3):824-828.
[12] SORRISO A,SORRENTINO P,RUCCO R,et al.An automated magnetoencephalographic data cleaning algorithm[J].Computer Methods in Biomechanics and Biomedical Engineering,2019,22(14):1116-1125.
[13] DALLACHIESA M,EBAID A,ELDAWY A,et al.NADEEF:a commodity data cleaning system[C]//Proceedings of 2013 ACM SIGMOD International Conference on Management of Data.New York,USA:ACM Press,2013:541-552.
[14] BATINI C,CAPPIELLO C,FRANCALANCI C,et al.Methodologies for data quality assessment and improvement[J].ACM Computing Surveys,2009,41(3):16-52.
[15] FENG Fujun,YAO Junping,LI Xiaojun.Research on the technology of data cleaning in big data[EB/OL].[2019-09-04].https://www.researchgate.net/publication/328918985_Research_on_the_Technology_of_Data_Cleaning_in_Big_Data.
[16] BESKALES G,ILYAS I F,GOLAB L,et al.On the relative trust between inconsistent data and inaccurate constraints[C]//Proceedings of the 29th International Conference on Data Engineering.Washington D.C.,USA:IEEE Press,2013:541-552.
[17] FAN Wenfei,LI Jianzhong,MA Shuai,et al.Interaction between record matching and data repairing[J].Journal of Data and Information Quality,2014,4(4):16-19.
[18] FAN W F,GEERTS F,TANG N,et al.Inferring data currency and consistency for conflict resolution[C]//Proceedings of the 29th International Conference on Data Engineering.Washington D.C.,USA:IEEE Press,2013:11-28.
[19] PAPOTTI P,XU C,ILYAS I F.Holistic data cleaning:putting violations into context[C]//Proceedings of 2013 IEEE International Conference on Data Engineering.Washington D.C.,USA:IEEE Press,2013:458-469.
[20] PENG Nan,WANG Hongya,DING Wencheng,et al.Finding interesting cleaning rules from dirty data[C]//Proceedings of the 10th International Symposium on Computational Intelligence and Design.Washington D.C.,USA:IEEE Press,2017:378-382.
[21] RIAHI I,MOUSSA F.A formal approach for modeling context-awarehuman-computer System[J].Computers and Electrical Engineering,2015,44:241-261.
[22] YUAN Chongyi.The application of Petri Net[M].Beijing:Science Press,2013.(in Chinese)袁崇义.Petri网应用[M].北京:科学出版社,2013.
[23] JIANG Wei,ZHOU Kaiqing,MO Liping.Parameter optimization strategy of fuzzy Petri Net utilizing hybrid GA-SFLA algorithm[M].Berlin,Germany:Springer,2019.
[24] LU Xing.Data cleaning technology and application of big data[J].Electronic Technology and Software Engineering,2019(9):157.(in Chinese)卢星.大数据的数据清洗技术及运用[J].电子技术与软件工程,2019(9):157.
[25] HE Jun,ZHANG Dehai.Big data intelligent analysis model of targeted poverty alleviation in Yunnan minority areas[J].Journal of Yunnan University for Nationalities(Natural Science Edition),2018,27(3):249-254.(in Chinese)何俊,张德海.云南少数民族地区精准扶贫大数据智能分析模型[J].云南民族大学学报(自然科学版),2018,27(3):249-254.
[26] NIE Yanmin.Using big data to improve poverty alleviation effect in deep poverty areas[J].People's Forum,2019(15):50-51.(in Chinese)聂燕敏.用大数据提升深度贫困地区脱贫成效[J].人民论坛,2019(15):50-51.

选择文件类型/文献管理软件名称

选择包含的内容

基于Petri网的数据清洗规则链自动组合与检测

Automatic Combination and Detection of Data Cleaning Rule Chains Based on Petri Net

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价

[1]	汪小勇, 董德存, 欧冬秀. 车载TBTC-CBTC系统降级场景下的CPN建模与仿真[J]. 计算机工程, 2022, 48(7): 189-198.
[2]	徐颖蕾, 马炳先. 无冲突Petri网的结构活性判定研究[J]. 计算机工程, 2021, 47(7): 296-300.
[3]	徐淑琳, 周广瑞, 岳昊. 标注Petri网中的最小初始标识估计[J]. 计算机工程, 2021, 47(4): 285-290,297.
[4]	马金林, 陈德光, 马自萍, 魏麟. 一种Petri网优化的验证码识别方法[J]. 计算机工程, 2020, 46(7): 277-285.
[5]	李联, 杨淏天. 基于GSPN的锁步处理器系统可靠性建模与分析[J]. 计算机工程, 2019, 45(7): 296-302.
[6]	李宗花, 叶正伟. 业务目标模型与业务场景模型的语义一致性分析[J]. 计算机工程, 2019, 45(12): 308-313.
[7]	陈莹,孙晓波,邢建春,杨启亮. 任务关键系统的时间约束验证与最优路径分析[J]. 计算机工程, 2018, 44(5): 60-65,77.
[8]	杜海森,杜玉越. 基于不完备日志的块状并发过程挖掘[J]. 计算机工程, 2018, 44(12): 56-61.
[9]	陈莹,邢建春,杨启亮,张孝鹏. 时间约束下任务关键系统的可调度性分析[J]. 计算机工程, 2018, 44(12): 115-119,128.
[10]	李堃,张雪松. STP安全通信协议设计与形式化验证[J]. 计算机工程, 2018, 44(12): 120-128.
[11]	冉德彤,游宏梁. 一种基于标签传播的数据分块算法[J]. 计算机工程, 2017, 43(9): 51-55,61.
[12]	张晶,王中正,范洪博. 基于随机Petri网的嵌入式软件能耗模型[J]. 计算机工程, 2017, 43(9): 316-321.
[13]	张晶,王亮,范洪博,肖智斌. 基于时间Petri网的嵌入式系统构件建模与能耗分析[J]. 计算机工程, 2017, 43(6): 30-34,39.
[14]	王雷,姜久雷,王晓峰. 基于Petri网的设计模式形式化描述[J]. 计算机工程, 2016, 42(7): 33-36,48.
[15]	陈龙,王立松. IMA重构的功能危害分析方法研究[J]. 计算机工程, 2016, 42(6): 151-155,160.

模态框（Modal）标题

选择文件类型/文献管理软件名称

选择包含的内容

基于Petri网的数据清洗规则链自动组合与检测

Automatic Combination and Detection of Data Cleaning Rule Chains Based on Petri Net

RichHTML

PDF

可视化

被引次数

摘要/Abstract

引用本文

使用本文

图/表 9

参考文献

相关文章 15

编辑推荐

Metrics

本文评价