摘要：

随着深度学习的高速发展, 基于序列到序列(Seq2Seq)架构的文本摘要方法成为研究焦点, 但现有大多数文本摘要模型受限于长期依赖, 忽略了注意力机制复杂度以及词序信息对文本摘要生成的影响, 生成的摘要丢失关键信息, 偏离原文内容与意图, 影响用户体验。为了解决上述问题, 提出一种基于Transformer改进的融合递归神经网络(RNN)与稀疏自注意力的文本摘要方法。首先采用窗口RNN模块, 将输入文本按窗口划分, 每个RNN对窗口内词序信息进行压缩, 并通过窗口级别的表示整合为整个文本的表示, 进而增强模型捕获局部依赖的能力; 其次采用基于递归循环机制的缓存模块, 循环缓存上一文本片段的信息到当前片段, 允许模型更好地捕获长期依赖和全局信息; 最后采用稀疏自注意力模块, 通过块稀疏矩阵对注意力矩阵按块划分, 关注并筛选出重要令牌对, 而不是在所有令牌对上平均分配注意力, 从而降低注意力的时间复杂度, 提高长文本摘要任务的效率。实验结果表明, 该方法在数据集text8、enwik8上的BPC分数相比于LoBART模型降低了0.02, 在数据集wikitext-103以及ptb上的PPL分数相比于LoBART模型分别降低了1.0以上, 验证了该方法的可行性与有效性。

关键词: 序列到序列架构, 文本摘要, Transformer模型, 递归神经网络, 递归循环机制, 稀疏自注意力机制

Abstract:

With the rapid development of deep learning, text summarization methods based on the Sequence-to-Sequence (Seq2Seq) architecture have been gaining traction. However, most existing text summarization models are constrained by long-term dependencies, ignore the complexity of attention mechanisms, and the impact of word-order information on text summary generation. The generated summaries often lose key information, deviating from the original content and intent, which affects user experience. To address these issues, a text summarization method based on Transformer improvements, combining a Recurrent Neural Network (RNN) with sparse self-attention, is proposed. First, a window-based RNN module is used to divide the input text into windows, compressing the word-order information within each window. These representations are then integrated at the window level to enhance the ability of the model to capture local dependencies. Second, a cache module based on recursive looping mechanisms is employed to recycle information from the previous text segment to the current segment, allowing the model to better capture long-term dependencies and global information. Finally, a sparse self-attention module is used to partition the attention matrix into blocks, focusing on and selecting important token pairs rather than evenly distributing attention across all token pairs. This reduces the time complexity of the attention and improves the efficiency of long-text summarization tasks. Experimental results show that compared to the LoBART model, this method reduces the BPC score by 0.02 on the text8 and enwik8 datasets and reduces the PPL score by more than 1.0 on the wikitext-103 and ptb datasets, validating the efficacy of this method.

Key words: Sequence-to-Sequence(Seq2Seq) architecture, text summarization, Transformer model, Recurrent Neural Network(RNN), recursive loop mechanism, sparse self-attention mechanism