Abstract: Most of the existing similarity calculation methods consider only the text structure features or the semantic information,and thus reduce the accuracy.To address the problem,this paper proposes a method,MCWFS,for calculating short text similarity using multi-check weighted fusion to deal with the sparse features of short texts.The method calculates the similarity by using three methods:similarity calculation based on improved edit distance,semantic similarity calculation considering word frequency,and similarity calculation based on the Word2vec and LSTM.Then weighted linear fusion is performed for texts that satisfy multi-check standards to avoid abnormal weighted similarity value caused by a single too large or too small similarity value.On this basis,weighted fusion is used to calculate the short text similarity to make the result more accurate and reasonable.Experimental results show that compared with the layer-by-layer check and non-check fusion methods,the proposed MCWFS method improves the average accuracy by 16.01% and 7.39% respectively,and its F1 value reaches 70.21%.

Key words: short text similarity, multi-check weighted fusion, edit distance, semantic information, word frequency

摘要： 传统相似度计算方法仅考虑文本结构特征或语义信息，从而导致准确率较低。结合短文本特征稀疏的特性，提出一种多重检验加权融合的短文本相似度计算方法MCWFS。使用基于改进编辑距离、考虑词频、基于Word2vec与LSTM的3种方法分别计算相似度，对满足多重检验标准的文本进行加权因子线性融合，以避免因一种相似度值过大或过小导致加权相似度值异常的问题。在此基础上，通过加权融合计算短文本相似度，使得计算结果更加准确合理。实验结果表明，相比层层检验和无检验融合方法，MCWFS方法的平均准确率分别提高16.01%和7.39%，且其F1值可达70.21%。

关键词: 短文本相似度, 多重检验加权融合, 编辑距离, 语义信息, 词频

CLC Number: 

• TP391