Triple Extraction Model for Legal Texts

doi:10.19678/j.issn.1000-3428.0057677

Abstract

Abstract: The open-source documents of criminal sentences on China judgments online contain important legal information.However,the documents are usually transcribed in the form of natural language and difficult for machines to understand.This paper proposes a triplet extraction model for legal texts to transform the unstructured texts recorded by natural language into structured triplets.In the construction of the model,the triplet extraction process is considered as a two-stage pipeline structure.The pretrained Bidirectional Encoder Representations from Transformer(BERT) model is used for Named Entity Recognition(NER),and the recognition results are applied to relation extraction to obtain the corresponding triplet representation,completing the information extraction for the unstructured legal texts of criminal senteces.Experimental results on the manually labeled dataset of criminal sentences show that the F1 score of the proposed model is 28.1 percentage points higher than that of combinational model based on recurrent neural network, demonstrating its excellent triplet extraction performance.

Key words: Named Entity Recognition(NER), relation extraction, pretrained language model, Transformer encoder, pipeline structure

摘要： 在中国裁判文书网上的开源刑事判决文档中蕴藏着重要的法律信息，但刑事判决书文档通常以自然语言的形式进行记录，而机器难以直接理解文档中的内容。为使由自然语言记录的非结构化刑事判决书文本转化为结构化三元组形式，构建一种面向法律文本的司法三元组抽取模型。将三元组抽取过程看作二阶段流水线结构，利用预训练的基于Transformer的双向编码器表示模型先进行命名实体识别，再将识别结果应用于关系抽取阶段得到相应的三元组表示，从而实现对非结构化刑事判决书文本的信息提取。实验结果表明，在经过人工标注的刑事判决书数据集上，该模型相比基于循环神经网络的组合模型的F1值提高了28.1个百分点，具有更优的三元组抽取性能。

关键词: 命名实体识别, 关系抽取, 预训练语言模型, Transformer编码器, 流水线结构

CLC Number:

TP391

CHEN Yanguang, WANG Lei, SUN Yuanyuan, WANG Zhizheng, ZHANG Shuchen. Triple Extraction Model for Legal Texts[J]. Computer Engineering, 2021, 47(5): 277-284.

陈彦光, 王雷, 孙媛媛, 王治政, 张书晨. 面向法律文本的三元组抽取模型[J]. 计算机工程, 2021, 47(5): 277-284.

/ / Recommend / Download Citations

URL: http://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0057677

http://www.ecice06.com/EN/Y2021/V47/I5/277

References

[1] WU Shanchan,HE Yifan.Enriching pre-trained language model with entity information for relation classification[C]//Proceedings of the 28th ACM International Conference on Information and Knowledge Management.New York,USA:ACM Press,2019:2361-2364.
[2] ISOZAKI H,KAZAWA H.Efficient support vector classifiers for named entity recognition[C]//Proceedings of the 19th International Conference on Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2002:1-7.
[3] BIKEL D M,MILLER S,SCHWARTZ R,et al.Nymble:a high-performance learning name-finder[C]//Proceedings of the 5th Conference on Applied Natural Language Processing.Philadelphia,USA:Association for Computational Linguistics,1997:194-201.
[4] LAFFERTY J D,MCCALLUM A,PEREIRA F C.Conditional random fields:probabilistic models for segmenting and labeling sequence data[C]//Proceedings of the 18th International Conference on Machine Learning.San Mateo,USA:Morgan Kaufmann Publishers Inc.,2001:282-289.
[5] LAMPLE G,BALLESTEROS M,SUBRAMANIAN S,et al.Neural architectures for named entity recognition[C]//Proceedings of 2016 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Philadelphia,USA:Association for Computational Linguistics,2016:260-270.
[6] ZHU Yuying,WANG Guoxin.CAN-NER:convolutional attention network for Chinese named entity recognition[C]//Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Philadelphia,USA:Association for Computational Linguistics,2019:3384-3393.
[7] ZHANG Yingcheng,YANG Yang,JIANG Rui,et al.Commercial intelligence entity recognition model based on BiLSTM-CRF[J].Computer Engineering,2019,45(5):308-314.(in Chinese)张应成,杨洋,蒋瑞,等.基于BiLSTM-CRF的商情实体识别模型[J].计算机工程,2019,45(5):308-314.
[8] DOZIER C,KONDADADI R,LIGHT M,et al.Named entity recognition and resolution in legal text[M].Berlin,Germany:Springer,2010.
[9] QUARESMA P,GONCALVES T.Using linguistic information and machine learning techniques to identify entities from juridical documents[M].Berlin,Germany:Springer,2010.
[10] HAQ M I U,LI Q,HASSAN S.Text mining techniques to capture facts for cloud computing adoption and big data processing[J].IEEE Access,2019,7:162254-162267.
[11] KAMBHATLA N.Combining lexical,syntactic,and semantic features with maximum entropy models for extracting relations[C]//Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2004:22-29.
[12] ZHOU Guodong,SU Jie,ZHANG Jie,et al.Exploring various knowledge in relation extraction[C]//Proceedings of the 43rd Annual Meeting on Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2005:427-434.
[13] CULOTTA A,SORENSEN J.Dependency tree kernels for relation extraction[C]//Proceedings of the 42nd Annual Meeting on Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2004:423-428.
[14] ZHOU Guodong,ZHANG Min,JI Donghong,et al.Tree kernel-based relation extraction with context-sensitive structured parse tree information[C]//Proceedings of 2007 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning.Prague,Czech Republic:[s.n.],2007:728-736.
[15] ZENG Daojian,LIU Kang,LAI Siwei,et al.Relation classification via convolutional deep neural network[C]//Proceedings of the 25th International Conference on Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2014:2335-2344.
[16] NGUYEN T H,GRISHMAN R.Relation extraction:perspective from convolutional neural networks[C]//Proceedings of the 1st Workshop on Vector Space Modeling for Natural Language Processing.Philadelphia,USA:Association for Computational Linguistics,2015:39-48.
[17] XIANG Bing,ZHOU Bowen.Classifying relations by ranking with convolutional neural networks[C]//Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing.Philadelphia,USA:Association for Computational Linguistics,2015:626-634.
[18] ZHANG Runyan,MENG Fanrong,ZHOU Yong,et al.Relation classification via recurrent neural network with attention and tensor layers[J].Big Data Mining and Analytics,2018,1(3):234-244.
[19] ZHOU Peng,SHI Wei,TIAN Jun,et al.Attention-based bidirectional long short-term memory networks for relation classification[C]//Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2016:207-212.
[20] SUN Ziyang,GU Junzhong,YANG Jing.Chinese entity relation extraction method based on deep learning[J].Computer Engineering,2018,44(9):164-170.(in Chinese)孙紫阳,顾君忠,杨静.基于深度学习的中文实体关系抽取方法[J].计算机工程,2018,44(9):164-170.
[21] VERGA P,STRUBELL E,MCCALLUM A.Simultaneously self-attending to all mentions for full-abstract biological relation extraction[C]//Proceedings of 2018 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Philadelphia,USA:Association for Computational Linguistics,2018:872-884.
[22] WANG H,TAN M,YU M,et al.Extracting multiple-relations in one-pass with pre-trained transformers[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2019:1371-1377.
[23] VASWANI A,SHAZEER N,PARMAR N,et al.Attention is all you need[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems.Long Beach,USA:Neural Information Processing Systems Foundation,Inc.,2017:6000-6010.
[24] DEVLIN J,CHANG M W,LEE K,et al.BERT:pre-training of deep bidirectional Transformers for language understanding[C]//Proceedings of 2019 Conference of the North American Chapter of the Association for Computational Linguistics:Human Language Technologies.Philadelphia,USA:Association for Computational Linguistics,2019:4171-4186.
[25] ALT C,HUBNER M,HENNIG L.Fine-tuning pre-trained transformer language models to distantly supervised relation extraction[C]//Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics.Philadelphia,USA:Association for Computational Linguistics,2019:1388-1398.
[26] PETERS M E,NEUMANN M,IYYER M,et al.Deep contextualized word representations[EB/OL].[2020-02-01].https://arxiv.org/abs/1802.05365.
[27] YANG Zhilin,DAI Zihang,YANG Yiming,et al.XLNet:generalized autoregressive pretraining for language understanding[C]//Proceedings of Advances in Neural Information Processing Systems.Vancouver,Canada:Neural Information Processing Systems Foundation,Inc.,2019:5754-5764.

Please choose a citation manager

Content to export