Combinatorial Generalization Ability Evaluation Method of SQL-to-text Model

doi:10.19678/j.issn.1000-3428.0067251

Abstract

Abstract:

Translating from Structured Query Language(SQL) to natural language can improve the usability of a database. Some progress is currently being made in this research, which mainly uses machine learning models. However, the capabilities of the existing translation models are still insufficient for practical applications. Because combinatorial generalization is a necessary ability for an SQL-to-text model to improve the translation effect in practical applications, and there is currently a lack of research on this ability for such models, a combination of SQL-to-text models is proposed as a generalization ability assessment method. This method generates a large amount of SQL and corresponding natural-language translations(referred to as SQL-natural language pairs) based on an existing SQL-to-text dataset. These SQL-natural language pairs are then divided into training and test data according to the number of SQL clauses they contain. Thus, the SQL clauses in the test data appear in the training data in different combinations, which produces a new data set that can be used to evaluate the generalization ability of the model combination. The evaluation results show that this method has a higher degree of query-knowledge use. It utilizes a more reasonable method to divide data, and the obtained data set meets the requirements for the evaluation of combinatorial generalization ability. It is close to the actual application scenario of the model, and is less restricted by the original data set. The combinatorial generalization ability of the existing models still needs to be further improved. Among them, the relationship-aware graph converter model designed for SQL-to-text tasks has the weakest combinatorial generalization ability, indicating that the original SQL-to-text data set is insufficient for the investigation of the combinatorial generalization ability.

Key words: Structured Query Language(SQL), compositional generalization, machine translation, database, Long Short-Term Memory(LSTM) model

摘要：

数据库的结构化查询语言（SQL）到自然语言的翻译(SQL-to-text)能提高关系数据库的易用性。近年来该领域主要使用机器学习的方法进行研究并已取得一定进展，然而现有翻译模型的能力仍不足以投入实际应用。由于组合泛化能力是SQL-to-text模型在实际应用中提升翻译效果的必要能力，且目前缺少对此类模型组合泛化能力的研究，因此提出一种SQL-to-text模型的组合泛化能力评估方法。基于现有的SQL-to-text数据集生成大量SQL和对应的自然语言翻译（SQL-自然语言对），并按SQL-自然语言对所含SQL子句的个数将其划分为训练数据与测试数据，使测试数据中的SQL子句皆以不同的组合方式在训练数据中出现，从而得到可评估模型组合泛化能力的新数据集。评估结果表明，该方法对查询知识的使用程度较高，划分数据的方式更加合理，所得数据集符合评估组合泛化能力的需求且贴近模型的实际应用场景，受到原始数据集的限制程度更低，并证实现有模型的组合泛化能力仍需提升，其中针对SQL-to-text任务设计的关系感知图转换器模型组合泛化能力最弱，表明原有的SQL-to-text数据集对组合泛化能力的考察存在欠缺。

关键词: 结构化查询语言, 组合泛化, 机器翻译, 数据库, 长短期记忆模型

Lin CHEN, Yuankai FAN, Zhenying HE, Xiaoqing LIU, Yang YANG, Lumin TANG. Combinatorial Generalization Ability Evaluation Method of SQL-to-text Model[J]. Computer Engineering, 2024, 50(3): 326-335.

陈琳, 范元凯, 何震瀛, 刘晓清, 杨阳, 汤路民. SQL-to-text模型的组合泛化能力评估方法[J]. 计算机工程, 2024, 50(3): 326-335.

/ Recommend / Download Citations

URL: https://www.ecice06.com/EN/10.19678/j.issn.1000-3428.0067251

https://www.ecice06.com/EN/Y2024/V50/I3/326

Figures/Tables 13

References 32

1	邓乃豪, 陈雨龙, 张岳. 根据自然语言生成SQL[EB/OL]. [2023-02-10]. https://mp.weixin.qq.com/s?__biz=MjM5MTY5ODE4OQ==&mid=2651534787&idx=2&sn=8df6ba7c99af22358d4e25b92033633d&chksm=bd4e09a18a3980b724e247a731875f2d4ec022edffa283d610bdf3e758af76637d844aa1ab2b&scene=27.
	DENG N H, CHEN Y L, ZHANG Y. Translate natural language to SQL[EB/OL]. [2023-02-10]. https://mp.weixin.qq.com/s?__biz=MjM5MTY5ODE4OQ==&mid=2651534787&idx=2&sn=8df6ba7c99af22358d4e25b92033633d&chksm=bd4e09a18a3980b724e247a731875f2d4ec022edffa283d610bdf3e758af76637d844aa1ab2b&scene=27. (in Chinese)
2	SIMITSIS A, IOANNIDIS Y. DBMSs should talk back too[EB/OL]. [2023-02-10]. https://arxiv.org/abs/0909.1786.pdf.
3	MA D, CHEN X Y, CAO R S, et al. Relation-aware graph transformer for SQL-to-text generation. Applied Sciences, 2021, 12(1): 369.
4	MONTAGUE R. Universal grammar. Theoria, 1970, 36(3): 373- 398. doi: 10.1111/j.1755-2567.1970.tb00434.x
5	LAKE B M, ULLMAN T D, TENENBAUM J B, et al. Building machines that learn and think like people. Behavioral and Brain Sciences, 2016, 40, 253- 267.
6	YU T, ZHANG R, YANG K, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2018: 3911-3921.
7	LAKE B, BARONI M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1711.00350v3.
8	KIM N, LINZEN T. COGS: a compositional generalization challenge based on semantic interpretation[EB/OL]. [2023-02-10]. https://arxiv.org/abs/2010.05465v1.
9	KEYSERS D, SCHARLI N, SCALES N, et al. Measuring compositional generalization: a comprehensive method on realistic data[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1912.09713.
10	BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of 2008 ACM SIGMOD International Conference on Management of Data. New York, USA: ACM Press, 2008: 224-236.
11	HUPKES D, DANKERS V, MUL M, et al. Compositionality decomposed: how do neural networks generalise?. Journal of Artificial Intelligence Research, 2020, 67, 757- 795. doi: 10.1613/jair.1.11674
12	PRICE P J. Evaluation of spoken language systems: the ATIS domain[C]//Proceedings of Workshop on Speech and Natural Language. New York, USA: ACM Press, 1990: 91-95.
13	DAHL D A, BATES M, BROWN M, et al. Expanding the scope of the ATIS task: the ATIS-3 corpus[C]//Proceedings of Workshop on Human Language Technology. New York, USA: ACM Press, 1994: 43-48.
14	ZELLE J, MOONEY R. Learning to parse database queries using inductive logic programming[C]//Proceedings of the 13th National Conference on Artificial Intelligence. New York, USA: ACM Press, 1996: 1050-1055.
15	TANG L R, MOONEY R J. Using multiple clause constructors in inductive logic programming for semantic parsing. Berlin, Germany: Springer, 2001.
16	POPESCU A M, ETZIONI O, KAUTZ H. Towards a theory of natural language interfaces to databases[C]//Proceedings of the 8th International Conference on Intelligent User Interfaces. New York, USA: ACM Press, 2003: 149-157.
17	IYER S, KONSTAS I, CHEUNG A, et al. Learning a neural semantic parser from user feedback[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Lisbon, Portugal: ACL Press, 2017: 963-973.
18	LI F, JAGADISH H V. Constructing an interactive natural language interface for relational databases. Proceedings of VLDB Endowment, 2014, 8(1): 73- 84. doi: 10.14778/2735461.2735468
19	FINEGAN-DOLLAK C, KUMMERFELD J K, ZHANG L, et al. Improving text-to-SQL evaluation methodology[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1806.09029.
20	ZHONG V, XIONG C M, SOCHER R. Seq2SQL: generating structured queries from natural language using reinforcement learning[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1709.00103.
21	GUO J Q, ZHAN Z, GAO Y, et al. Towards complex text-to-SQL in cross-domain database with intermediate representation[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1905.08205v2.
22	WANG B L, SHIN R, LIU X D, et al. RAT-SQL: relation-aware schema encoding and linking for text-to-SQL parsers[EB/OL]. [2023-02-10]. http://arxiv.org/abs/1911.04942v4.
23	ZHANG A, WU K, WANG L J, et al. Data augmentation with hierarchical SQL-to-question generation for cross-domain text-to-SQL parsing[C]//Proceedings of EMNLPʼ21. Washington D. C., USA: IEEE Press, 2021: 8974-898.
24	SHAW P, CHANG M W, PASUPAT P, et al. Compositional generalization and natural language variation: can a semantic parsing approach handle both?[EB/OL]. [2023-02-10]. http://arxiv.org/abs/2010.12725v2.
25	HOCHREITER S, SCHMIDHUBER J. Long short-term memory. Neural Computation, 1997, 9(8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735
26	GRAVES A, SCHMIDHUBER J. Framewise phoneme classification with bidirectional LSTM networks[C]//Proceedings of IEEE International Joint Conference on Neural Networks. Washington D. C., USA: IEEE Press, 2005: 2047-2052.
27	VASWANI A, SHAZEER N M, PARMAR N, et al. Attention is all you need[C]//Proceedings of NIPSʼ17. Cambridge, USA: MIT Press, 2017: 5998-6008.
28	SHAW P, USZKOREIT J, VASWANI A. Self-attention with relative position representations[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1803.02155.
29	TAI K S, SOCHER R, MANNING C D. Improved semantic representations from tree-structured long short-term memory networks[EB/OL]. [2023-02-10]. https://arxiv.org/pdf/1503.00075.pdf.
30	赵志超, 游进国, 何培蕾, 等. 数据库中文查询对偶学习式生成SQL语句研究. 中文信息学报, 2023, 37(3): 164- 172. doi: 10.3969/j.issn.1003-0077.2023.03.016
	ZHAO Z C, YOU J G, HE P L, et al. Generating SQL statement from Chinese query based on dual learning. Journal of Chinese Information Processing, 2023, 37(3): 164- 172. doi: 10.3969/j.issn.1003-0077.2023.03.016
31	赵猛, 陈珂, 寿黎但, 等. 基于树状模型的复杂自然语言查询转SQL技术研究. 软件学报, 2022, 33(12): 4727- 4745. URL
	ZHAO M, CHEN K, SHOU L D, et al. Converting complex natural language query to SQL based on tree representation model. Journal of Software, 2022, 33(12): 4727- 4745. URL
32	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. [S. 1. ]: ACL Press, 2002: 311-318.

[1]	MO Shaocong, CHEN Qingfeng, XIE Ze, LIU Chunyu, QIU Junlai. Entity Alignment Based on Dynamic Graph Attention and Label Propagation [J]. Computer Engineering, 2024, 50(4): 150-159.
[2]	HOU Yutao, Abudukelimu Abulizi, SHI Yaqing, Mayilamu Musideke, Halidanmu Abudukelimu. Research on Low-Resource Language Machine Translation for the ″Belt and Road″ [J]. Computer Engineering, 2024, 50(4): 332-341.
[3]	Halidanmu ABUDUKELIMU, Yutao HOU, Dengfeng YAO, Abudukelimu ABULIZI, Jishang CHEN. Survey of Uyghur Machine Translation Research [J]. Computer Engineering, 2024, 50(1): 1-16.
[4]	Xingxing DONG, Jixun GAO, Xiaotong WANG, Song LI. Survey of Research on Representation and Reasoning Models of Spatial Directional Relations [J]. Computer Engineering, 2023, 49(9): 1-15.
[5]	Jiading GUO, Peng WANG. Graph Query Processing Technology Based on Data Warehouse [J]. Computer Engineering, 2023, 49(9): 32-42.
[6]	RONG Keyao, XIONG Yun. Automatic Code Annotation Generation Based on Multi-dimensional Heterogeneous Graph Structure [J]. Computer Engineering, 2023, 49(4): 240-248.
[7]	Jinpeng ZHANG, Xiangyu DUAN. Terminology Intervention Translation Model Combining Vectorization Method and Mask Mechanism [J]. Computer Engineering, 2023, 49(11): 70-76, 84.
[8]	Renchong DUAN, Xiangyu DUAN. Neural Machine Translation Based on Adaptive Training and Drop Mechanism [J]. Computer Engineering, 2023, 49(10): 120-126, 135.
[9]	CUI Weiqi, YAN Xin, TENG Lei, CHEN Wei, XU Guangyi. A Method for Improving Performance of Opinion Targets Extraction by Evaluating Category Classification [J]. Computer Engineering, 2022, 48(11): 96-103,136.
[10]	ZHU Wenjun, XU Zhuang, QIN Jiajia, LI Peng. DPDK-Based Optimization Approach for High Speed Storage I/O [J]. Computer Engineering, 2021, 47(7): 205-211,217.
[11]	QIAN Shangyun, SHAO Zhiyuan, ZHENG Ran, CHEN Jilin. GPU-based Graph Analysis and Computation Method for Graph Database [J]. Computer Engineering, 2021, 47(6): 52-59.
[12]	FU Youjia. Facial Pose Estimation Method on Single Image Based on Facial Feature Points [J]. Computer Engineering, 2021, 47(4): 197-203,210.
[13]	WANG Zhiduo, JIANG Bo, MIAO Rui, ZHAO Hui. Design and Implementation of Solution Algorithm for Foreign Key Conflict Based on Directed Graph [J]. Computer Engineering, 2021, 47(2): 254-260.
[14]	CHEN Xi, YANG Yating, DONG Rui. Research on BERT Embedding for Chinese-Uyghur Machine Translation [J]. Computer Engineering, 2021, 47(12): 112-117.
[15]	WANG Haoliang, LIAN Yuzhong, WANG Lili. Research and Implementation of Blockchain Technology Scheme for Electronic License Sharing [J]. Computer Engineering, 2020, 46(8): 277-283.

Please choose a citation manager

Content to export