摘要：

文本相似度计算是自然语言处理的一部分, 用来计算两个词、句子及文本之间的相似程度, 具有多种应用场景, 文本相似度计算的研究对于人工智能的发展有着重要作用。文本相似度计算起初基于字符串表面, 随着词向量的提出, 文本相似度计算可进行基于统计以及深度学习的建模与计算, 也可与预训练模型相结合。首先, 将文本相似度计算方法分为基于字符串、基于词向量、基于预训练模型、基于深度学习、其他方法5类, 并对这些方法进行简要介绍。然后, 根据不同文本相似度计算方法的原理, 具体介绍了编辑距离、汉明距离、词袋模型、向量空间模型(VSM)、深度结构语义模型(DSSM)、句子嵌入的简单对比学习(SimCSE)等常见方法。最后, 对文本相似度计算常用的数据集以及评价标准进行整理和分析, 并对文本相似度计算的未来发展进行展望。

关键词: 文本相似度, 字符串, 词向量, 预训练模型, 深度学习

Abstract:

Text similarity calculation is a part of natural language processing and is used to calculate the similarity between two words, sentences, or texts in many application scenarios. Research on text similarity calculation plays an important role in the development of artificial intelligence. Text similarity calculation has conventionally been based on character string surfaces. With the introduction of word vectors, text similarity calculation can be modeled and calculated based on statistics and deep learning, in addition to combining it with pre-trained models. First, text similarity calculation methods can be divided into five categories: character string-based, word vector-based, pre-trained model-based, deep learning-based, and other methods. Each category is briefly introduced. Subsequently, according to the principles of the different text similarity calculation methods, common methods such as the edit distance, Hamming distance, bag of words model, Vector Space Model (VSM), Deep Structured Semantic Model (DSSM), and Simple Contrastive learning of Sentence Embedding (SimCSE) are discussed. Finally, commonly used data sets and evaluation criteria for text similarity calculation are sorted and analyzed, and the future development of text similarity calculation is prospected.

Key words: text similarity, character string, word vector, pre-trained model, deep learning