Abstract: While punctuation restoration of standardized text has made significant progress, real-time punctuation restoration within Automatic Speech Recognition (ASR) systems remains a challenging task. The core challenge lies in the colloquial nature of ASR transcripts and the severe imbalance of punctuation categories. Moreover, despite strong semantic understanding capabilities, large language models (LLMs) suffer from high inference latency and deployment costs. To alleviate these problems, this paper proposes a real-time punctuation restoration method based on progressive iterative optimization. First, multiple public Chinese corpora are integrated to construct a large-scale training dataset covering diverse text domains. In addition, a punctuation-aware data weighting strategy is introduced to mitigate the learning bias caused by imbalanced punctuation distributions. By assigning different weights to training samples, the proposed strategy improves the alignment between the training corpus and real-world ASR outputs. Then, by introducing dynamic masking to restrict the attention range, only the context information within a limited window near the current position is retained, ensuring the ability to obtain key semantic information and effectively reducing long-distance noise interference, thereby achieving a balance between accuracy and latency. Finally, a progressive iterative optimization mechanism is introduced. The sliding-window average loss is used to estimate sample difficulty, and a dynamic weighting strategy is designed accordingly. For difficult samples that continuously generate high prediction losses, a multi-sentence concatenation method is used to generate new samples with more complex context dependencies, and their training weights are increased, enabling the model to gradually focus on complex semantic boundaries, achieving the collaborative evolution of training data and model capabilities, and thereby continuously improving the model's ability to capture complex semantic structures. This paper conducts systematic experiments on the real punctuation restoration test set and compares it with representative models such as CT-transformer, Qwen2.5-7B, Llama3.1-8B, Gemma2-9B, and DeepSeek-V3. The experimental results show that the F1 value of this method is 4.92 percentage points higher than that of the traditional real-time punctuation restoration baseline CT-transformer model. Moreover, when compared with large language models with parameter scales much larger than this model, this method still achieves better restoration performance. In terms of inference efficiency, the average inference time per sentence of the model is only 26 ms, which is nearly 200 times faster than DeepSeek-V3, meeting the requirements of low latency response in real-time speech processing scenarios. Ablation experiments further verify the effectiveness of the controllable delay mechanism, data weighting strategy, and progressive iterative optimization module in improving model performance. Among them, the iterative optimization process gradually increases the F1 value from 38.78% to 43.70%. Compared with general large models relying on large-scale parameters, the data and model collaborative optimization scheme designed based on task characteristics can achieve better punctuation restoration effects while maintaining low resource consumption and low-latency inference, demonstrating strong practical value for real-time ASR post-processing applications.

摘要： 目前规范文本的标点恢复已经取得了很好的结果，但是语音识别系统中的实时标点恢复任务仍然面临挑战。其核心挑战为真实语音识别输出往往包含口语化表达和类别分布不均衡等特点，导致传统分类模型难以准确识别复杂语义边界，而生成式大模型虽然具有较强语义理解能力，却存在推理延迟高、部署成本大等问题，难以满足实时应用需求。为缓解上述问题，本文提出一种基于渐进式迭代优化的实时标点恢复方法。首先，通过整合多个公开中文语料库，构建覆盖多种类型的大规模训练语料。同时，设计了一种标点感知的数据加权策略，通过改变不同样本的权重缓解逗号、句号和问号等类别分布不均衡带来的学习偏差，实现训练数据与真实语音识别输出文本的有效对齐。然后，通过动态掩码限制注意力计算范围，仅保留当前位置附近有限窗口内的上下文信息，保证关键语义信息获取能力并有效减少远距离噪声干扰，从而实现精度与延迟之间的平衡。最后，提出一种渐进式迭代优化机制，通过记录训练过程中样本的滑动窗口平均损失，对训练样本进行难度评估，并构建动态权重更新策略。对于持续产生较高预测损失的困难样本，采用多句拼接方式生成具有更复杂上下文依赖关系的新样本，并提高其训练权重，使模型逐步聚焦于复杂语义边界，实现训练数据与模型能力的协同演化，从而不断提升模型对复杂语义结构的建模能力。本文在真实标点恢复测试集上进行了系统实验，并与CT-transformer以及Qwen2.5-7B、Llama3.1-8B、Gemma2-9B、DeepSeek-V3等代表性模型进行比较。实验结果表明，本文方法相较于传统实时标点恢复基线CT-transformer模型的 F1值提升4.92个百分点。同时，在与参数规模远大于本模型的大语言模型比较中，本文方法仍取得更优的恢复性能。在推理效率方面，模型单句平均推理时间仅为26 ms，较DeepSeek-V3提升近200倍，满足实时语音处理场景对低延迟响应的要求。消融实验进一步验证了可控时延机制、数据加权策略以及渐进式迭代优化模块对模型性能提升的有效性，其中迭代优化过程使模型F1值由38.78%逐步提升至43.70%。相比依赖大规模参数的通用大模型，针对任务特征设计的数据与模型协同优化方案能够在保持低资源消耗和低延迟推理的前提下获得更优的标点恢复效果，为实时语音识别后处理系统的工程部署提供了一种兼顾精度、效率与可扩展性的解决方案。