SQL-to-text模型的组合泛化能力评估方法

doi:10.19678/j.issn.1000-3428.0067251

摘要/Abstract

摘要：

数据库的结构化查询语言（SQL）到自然语言的翻译(SQL-to-text)能提高关系数据库的易用性。近年来该领域主要使用机器学习的方法进行研究并已取得一定进展，然而现有翻译模型的能力仍不足以投入实际应用。由于组合泛化能力是SQL-to-text模型在实际应用中提升翻译效果的必要能力，且目前缺少对此类模型组合泛化能力的研究，因此提出一种SQL-to-text模型的组合泛化能力评估方法。基于现有的SQL-to-text数据集生成大量SQL和对应的自然语言翻译（SQL-自然语言对），并按SQL-自然语言对所含SQL子句的个数将其划分为训练数据与测试数据，使测试数据中的SQL子句皆以不同的组合方式在训练数据中出现，从而得到可评估模型组合泛化能力的新数据集。评估结果表明，该方法对查询知识的使用程度较高，划分数据的方式更加合理，所得数据集符合评估组合泛化能力的需求且贴近模型的实际应用场景，受到原始数据集的限制程度更低，并证实现有模型的组合泛化能力仍需提升，其中针对SQL-to-text任务设计的关系感知图转换器模型组合泛化能力最弱，表明原有的SQL-to-text数据集对组合泛化能力的考察存在欠缺。

关键词: 结构化查询语言, 组合泛化, 机器翻译, 数据库, 长短期记忆模型

Abstract:

Translating from Structured Query Language(SQL) to natural language can improve the usability of a database. Some progress is currently being made in this research, which mainly uses machine learning models. However, the capabilities of the existing translation models are still insufficient for practical applications. Because combinatorial generalization is a necessary ability for an SQL-to-text model to improve the translation effect in practical applications, and there is currently a lack of research on this ability for such models, a combination of SQL-to-text models is proposed as a generalization ability assessment method. This method generates a large amount of SQL and corresponding natural-language translations(referred to as SQL-natural language pairs) based on an existing SQL-to-text dataset. These SQL-natural language pairs are then divided into training and test data according to the number of SQL clauses they contain. Thus, the SQL clauses in the test data appear in the training data in different combinations, which produces a new data set that can be used to evaluate the generalization ability of the model combination. The evaluation results show that this method has a higher degree of query-knowledge use. It utilizes a more reasonable method to divide data, and the obtained data set meets the requirements for the evaluation of combinatorial generalization ability. It is close to the actual application scenario of the model, and is less restricted by the original data set. The combinatorial generalization ability of the existing models still needs to be further improved. Among them, the relationship-aware graph converter model designed for SQL-to-text tasks has the weakest combinatorial generalization ability, indicating that the original SQL-to-text data set is insufficient for the investigation of the combinatorial generalization ability.

Key words: Structured Query Language(SQL), compositional generalization, machine translation, database, Long Short-Term Memory(LSTM) model

陈琳, 范元凯, 何震瀛, 刘晓清, 杨阳, 汤路民. SQL-to-text模型的组合泛化能力评估方法[J]. 计算机工程, 2024, 50(3): 326-335.

Lin CHEN, Yuankai FAN, Zhenying HE, Xiaoqing LIU, Yang YANG, Lumin TANG. Combinatorial Generalization Ability Evaluation Method of SQL-to-text Model[J]. Computer Engineering, 2024, 50(3): 326-335.

https://www.ecice06.com/CN/Y2024/V50/I3/326

图/表 13

图1 SQL-to-text模型组合泛化能力评估方法整体框架

Fig.1 Overall framework of compositional generalization ability evaluation method for SQL-to-text models

图2 关键字标注示例

Fig.2 Example of keyword tagging

参考文献 32

1	邓乃豪, 陈雨龙, 张岳. 根据自然语言生成SQL[EB/OL]. [2023-02-10]. https://mp.weixin.qq.com/s?__biz=MjM5MTY5ODE4OQ==&mid=2651534787&idx=2&sn=8df6ba7c99af22358d4e25b92033633d&chksm=bd4e09a18a3980b724e247a731875f2d4ec022edffa283d610bdf3e758af76637d844aa1ab2b&scene=27.
	DENG N H, CHEN Y L, ZHANG Y. Translate natural language to SQL[EB/OL]. [2023-02-10]. https://mp.weixin.qq.com/s?__biz=MjM5MTY5ODE4OQ==&mid=2651534787&idx=2&sn=8df6ba7c99af22358d4e25b92033633d&chksm=bd4e09a18a3980b724e247a731875f2d4ec022edffa283d610bdf3e758af76637d844aa1ab2b&scene=27. (in Chinese)
2	SIMITSIS A, IOANNIDIS Y. DBMSs should talk back too[EB/OL]. [2023-02-10]. https://arxiv.org/abs/0909.1786.pdf.
3	MA D, CHEN X Y, CAO R S, et al. Relation-aware graph transformer for SQL-to-text generation. Applied Sciences, 2021, 12(1): 369.
4	MONTAGUE R. Universal grammar. Theoria, 1970, 36(3): 373- 398. doi: 10.1111/j.1755-2567.1970.tb00434.x
5	LAKE B M, ULLMAN T D, TENENBAUM J B, et al. Building machines that learn and think like people. Behavioral and Brain Sciences, 2016, 40, 253- 267.
6	YU T, ZHANG R, YANG K, et al. Spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task[C]//Proceedings of 2018 Conference on Empirical Methods in Natural Language Processing. Stroudsburg, USA: Association for Computational Linguistics, 2018: 3911-3921.
7	LAKE B, BARONI M. Generalization without systematicity: on the compositional skills of sequence-to-sequence recurrent networks[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1711.00350v3.
8	KIM N, LINZEN T. COGS: a compositional generalization challenge based on semantic interpretation[EB/OL]. [2023-02-10]. https://arxiv.org/abs/2010.05465v1.
9	KEYSERS D, SCHARLI N, SCALES N, et al. Measuring compositional generalization: a comprehensive method on realistic data[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1912.09713.
10	BOLLACKER K, EVANS C, PARITOSH P, et al. Freebase: a collaboratively created graph database for structuring human knowledge[C]//Proceedings of 2008 ACM SIGMOD International Conference on Management of Data. New York, USA: ACM Press, 2008: 224-236.
11	HUPKES D, DANKERS V, MUL M, et al. Compositionality decomposed: how do neural networks generalise?. Journal of Artificial Intelligence Research, 2020, 67, 757- 795. doi: 10.1613/jair.1.11674
12	PRICE P J. Evaluation of spoken language systems: the ATIS domain[C]//Proceedings of Workshop on Speech and Natural Language. New York, USA: ACM Press, 1990: 91-95.
13	DAHL D A, BATES M, BROWN M, et al. Expanding the scope of the ATIS task: the ATIS-3 corpus[C]//Proceedings of Workshop on Human Language Technology. New York, USA: ACM Press, 1994: 43-48.
14	ZELLE J, MOONEY R. Learning to parse database queries using inductive logic programming[C]//Proceedings of the 13th National Conference on Artificial Intelligence. New York, USA: ACM Press, 1996: 1050-1055.
15	TANG L R, MOONEY R J. Using multiple clause constructors in inductive logic programming for semantic parsing. Berlin, Germany: Springer, 2001.
16	POPESCU A M, ETZIONI O, KAUTZ H. Towards a theory of natural language interfaces to databases[C]//Proceedings of the 8th International Conference on Intelligent User Interfaces. New York, USA: ACM Press, 2003: 149-157.
17	IYER S, KONSTAS I, CHEUNG A, et al. Learning a neural semantic parser from user feedback[C]//Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics. Lisbon, Portugal: ACL Press, 2017: 963-973.
18	LI F, JAGADISH H V. Constructing an interactive natural language interface for relational databases. Proceedings of VLDB Endowment, 2014, 8(1): 73- 84. doi: 10.14778/2735461.2735468
19	FINEGAN-DOLLAK C, KUMMERFELD J K, ZHANG L, et al. Improving text-to-SQL evaluation methodology[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1806.09029.
20	ZHONG V, XIONG C M, SOCHER R. Seq2SQL: generating structured queries from natural language using reinforcement learning[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1709.00103.
21	GUO J Q, ZHAN Z, GAO Y, et al. Towards complex text-to-SQL in cross-domain database with intermediate representation[EB/OL]. [2023-02-10]. https://arxiv.org/abs/1905.08205v2.
22	WANG B L, SHIN R, LIU X D, et al. RAT-SQL: relation-aware schema encoding and linking for text-to-SQL parsers[EB/OL]. [2023-02-10]. http://arxiv.org/abs/1911.04942v4.
23	ZHANG A, WU K, WANG L J, et al. Data augmentation with hierarchical SQL-to-question generation for cross-domain text-to-SQL parsing[C]//Proceedings of EMNLPʼ21. Washington D. C., USA: IEEE Press, 2021: 8974-898.
24	SHAW P, CHANG M W, PASUPAT P, et al. Compositional generalization and natural language variation: can a semantic parsing approach handle both?[EB/OL]. [2023-02-10]. http://arxiv.org/abs/2010.12725v2.
25	HOCHREITER S, SCHMIDHUBER J. Long short-term memory. Neural Computation, 1997, 9(8): 1735- 1780. doi: 10.1162/neco.1997.9.8.1735
26	GRAVES A, SCHMIDHUBER J. Framewise phoneme classification with bidirectional LSTM networks[C]//Proceedings of IEEE International Joint Conference on Neural Networks. Washington D. C., USA: IEEE Press, 2005: 2047-2052.
27	VASWANI A, SHAZEER N M, PARMAR N, et al. Attention is all you need[C]//Proceedings of NIPSʼ17. Cambridge, USA: MIT Press, 2017: 5998-6008.
28	SHAW P, USZKOREIT J, VASWANI A. Self-attention with relative position representations[EB/OL]. [2023-02-10]. http://arxiv.org/pdf/1803.02155.
29	TAI K S, SOCHER R, MANNING C D. Improved semantic representations from tree-structured long short-term memory networks[EB/OL]. [2023-02-10]. https://arxiv.org/pdf/1503.00075.pdf.
30	赵志超, 游进国, 何培蕾, 等. 数据库中文查询对偶学习式生成SQL语句研究. 中文信息学报, 2023, 37(3): 164- 172. doi: 10.3969/j.issn.1003-0077.2023.03.016
	ZHAO Z C, YOU J G, HE P L, et al. Generating SQL statement from Chinese query based on dual learning. Journal of Chinese Information Processing, 2023, 37(3): 164- 172. doi: 10.3969/j.issn.1003-0077.2023.03.016
31	赵猛, 陈珂, 寿黎但, 等. 基于树状模型的复杂自然语言查询转SQL技术研究. 软件学报, 2022, 33(12): 4727- 4745. URL
	ZHAO M, CHEN K, SHOU L D, et al. Converting complex natural language query to SQL based on tree representation model. Journal of Software, 2022, 33(12): 4727- 4745. URL
32	PAPINENI K, ROUKOS S, WARD T, et al. BLEU: a method for automatic evaluation of machine translation[C]//Proceedings of the 40th Annual Meeting on Association for Computational Linguistics. [S. 1. ]: ACL Press, 2002: 311-318.

[1]	张亚洲, 和玉, 戎璐, 王祥凯. 基于上下文知识增强型Transformer网络的抑郁检测[J]. 计算机工程, 2024, 50(8): 75-85.
[2]	莫少聪, 陈庆锋, 谢泽, 刘春雨, 邱俊铼. 基于动态图注意力与标签传播的实体对齐[J]. 计算机工程, 2024, 50(4): 150-159.
[3]	侯钰涛, 阿布都克力木·阿布力孜, 史亚庆, 马依拉木·木斯得克, 哈里旦木·阿布都克里木. 面向"一带一路"的低资源语言机器翻译研究[J]. 计算机工程, 2024, 50(4): 332-341.
[4]	哈里旦木·阿布都克里木, 侯钰涛, 姚登峰, 阿布都克力木·阿布力孜, 陈吉尚. 维吾尔语机器翻译研究综述[J]. 计算机工程, 2024, 50(1): 1-16.
[5]	董星星, 高继勋, 王晓桐, 李松. 空间方向关系表达与推理模型研究综述[J]. 计算机工程, 2023, 49(9): 1-15.
[6]	郭家鼎, 王鹏. 基于数据仓库的典型图查询处理技术[J]. 计算机工程, 2023, 49(9): 32-42.
[7]	戎珂瑶, 熊贇. 基于多维度异质图结构的代码注释自动生成[J]. 计算机工程, 2023, 49(4): 240-248.
[8]	张金鹏, 段湘煜. 结合向量化方法与掩码机制的术语干预翻译模型[J]. 计算机工程, 2023, 49(11): 70-76, 84.
[9]	段仁翀, 段湘煜. 基于适应性训练与丢弃机制的神经机器翻译[J]. 计算机工程, 2023, 49(10): 120-126, 135.
[10]	黄君扬, 王振宇, 梁家卿, 肖仰华. 基于自裁剪异构图的NL2SQL模型[J]. 计算机工程, 2022, 48(9): 71-77,88.
[11]	崔伟琪, 严馨, 滕磊, 陈玮, 徐广义. 一种通过评价类别分类提升评价对象抽取性能的方法[J]. 计算机工程, 2022, 48(11): 96-103,136.
[12]	朱文俊, 徐壮, 秦家佳, 李鹏. 基于DPDK的高速存储I/O优化方法[J]. 计算机工程, 2021, 47(7): 205-211,217.
[13]	钱裳云, 邵志远, 郑然, 陈继林. 图数据库中基于GPU的图分析计算方法[J]. 计算机工程, 2021, 47(6): 52-59.
[14]	傅由甲. 基于面部特征点的单幅图像人脸姿态估计方法[J]. 计算机工程, 2021, 47(4): 197-203,210.
[15]	王智铎, 江波, 苗瑞, 赵慧. 基于有向图的外键冲突解决算法设计与实现[J]. 计算机工程, 2021, 47(2): 254-260.

选择文件类型/文献管理软件名称

选择包含的内容