Abstract: With the rapid development of Large Language Model (LLM), LLM has shown promising application prospects in text classification tasks. Existing text classification methods based on prompt engineering heavily rely on the generative capability of LLM, and often suffer from insufficient generation ability and poor reasoning stability in lightweight LLM scenarios, making it difficult to achieve satisfactory performance. In addition, different hidden layers of LLM exhibit significant differences in representation capability, and intermediate layer representations often contain richer semantic information, even outperforming the last layer outputs in certain tasks. To address these issues, a multi-level semantic fusion for text classification based on LLM is proposed. By extracting and fusing text embedding representations from different hidden layers of lightweight LLM, the proposed method achieves efficient and stable classification performance without fine-tuning LLM. Firstly, prompt engineering is introduced into the construction process of text embedding representations. By designing prompt templates suitable for text classification tasks, original sentences are inserted into specific contextual semantics to guide the model in adjusting the attention distribution when generating hidden layer representations, thereby enhancing the discriminative capability of text embedding representations. Secondly, curvature is employed to characterize the geometric smoothness of hidden layer representations. Hidden layer representations with lower curvature correspond to smoother semantic trajectories and are more suitable as text embedding representations for text classification tasks. By calculating the average curvature of each hidden layer, several hidden layer representations with lower curvature are selected as feature sources for subsequent fusion. Thirdly, intra-layer position-weighted pooling and inter-layer multi-scale convolution fusion are used to obtain more expressive sentence embeddings. The intra-layer position-weighted pooling assigns different weights according to the relative positions of tokens in the input sequence, preserving important semantic information at the tail while also considering contextual information at the beginning of the sequence, thereby generating more complete and stable text embedding representations. The inter-layer multi-scale convolution fusion models relationships among different hidden layers through convolution operations to achieve the fusion of multi-level semantic information. Finally, the classifier is trained using a joint loss function, simultaneously improving classification accuracy, the learning capability for hard samples, and the discriminative of the representation space, thereby optimizing model parameters from multiple perspectives to obtain the final prediction results. Experimental results demonstrate that the proposed method achieves accuracies of 96.11%, 93.48%, 89.92%, 74.15%, and 96.33% on the SST-2, MR, 20NG, Ohsumed, and R52 datasets, respectively, outperforming existing mainstream baseline methods. In addition, Macro-F1 is introduced as a supplementary evaluation metric on the relatively imbalanced Ohsumed and R52 datasets, where the proposed method achieves 62.93% and 73.12%, respectively, demonstrating its effectiveness under complex data distribution conditions. Furthermore, ablation studies, different intra-layer pooling methods analysis, different prompt templates analysis, and different numbers of layers analysis are conducted to verify the rationality and effectiveness of each module design. Sensitivity analysis of loss weights is performed to evaluate the influence of different loss weight variations on classification performance. Different layers performance analysis is performed to explore the relationship between different hidden layer representations and classification performance. In conclusion, the proposed method can effectively extract and fuse LLM embedding representations for text classification, significantly reducing computational overhead while improving the representation quality and discriminative capability, demonstrating strong generalizability and practical applicability.

摘要： 随着大语言模型（LLM）的快速发展，LLM在文本分类任务中展现出良好的应用前景。现有基于提示词工程的文本分类方法高度依赖LLM的生成能力，在轻量级LLM场景下存在生成能力不足、推理稳定性差的问题，往往难以取得理想效果。此外，LLM不同隐藏层在表征能力上存在显著差异，其中间层表示往往蕴含丰富的语义信息，甚至在部分任务中优于其最后一层的输出。针对上述问题，提出一种基于LLM的多层次语义融合文本分类方法，通过提取并融合轻量级LLM不同隐藏层的文本嵌入表示，在无需对LLM进行参数微调的条件下，实现高效稳定的分类效果。首先，将提示词工程引入文本嵌入表示构建过程，通过设计适用于文本分类任务的提示模板，将原始句子插入特定上下文语义中，以引导模型在生成隐藏层表示时调整句向量的注意力分布，影响文本嵌入表示的判别能力。其次，使用曲率刻画隐藏层表示的几何变化平滑性，曲率较低的隐藏层表示对应更平滑的语义轨迹，更适合作为文本嵌入表示用于文本分类任务中。通过计算各隐藏层的层平均曲率，并选取曲率较低的若干个隐藏层表示作为后续融合的特征来源。再次，通过层内位置加权融合和层间多尺度卷积融合，获得更具表达能力的文本嵌入表示，层内位置加权融合根据词元在输入序列中的相对位置分配不同权重，在保留尾部重要语义信息的同时兼顾前部上下文内容，以获得更加完整稳定的文本嵌入表示。层间多尺度卷积融合通过卷积操作建模不同隐藏层之间的语义关联关系，实现多层语义信息的融合。最后，联合多种损失函数训练分类器，同时提升分类准确性、较难分类样本学习能力以及表示空间的判别性，从不同角度对参数进行优化训练实现最终结果预测。实验结果表明，所提出的方法在SST-2、MR、20NG、Ohsumed和R52五个数据集上的准确率分别达到96.11%、93.48%、89.92%、74.15%和96.33%，优于现有主流基线方法；在Ohsumed与R52两个类别分布相对不均衡的数据集上，引入宏平均F1分数作为补充评价指标，结果分别达到62.93%和73.12%，验证其在复杂数据分布条件下的分类能力。进一步地，通过消融实验、不同层内融合方法效果分析、不同提示模板效果分析、不同数量层数融合效果分析，充分验证各模块设计的合理性与有效性；通过损失函数权重敏感性分析，评估分类性能对不同损失函数权重变化的敏感程度；通过不同层效果分析，探讨不同隐藏层表示与分类性能之间的关系。综上所述，所提出方法能够有效提取并融合LLM的嵌入表示用于文本分类，大幅降低计算开销的同时提升文本的表示能力与判别能力，具有良好的泛化能力和实际应用价值。