Abstract:

Large Language Models (LLM) have demonstrated outstanding performance in natural language processing tasks. However, their extremely large parameter scales pose a significant challenge because the limited capacity of GPU memory becomes a performance bottleneck for inference tasks. To address this issue in the context of LLM inference services, this study proposes AdaptiveLLM, which enables the adaptive selection of offloading strategies between tensor swapping and tensor recomputation based on the characteristics of inference task workloads. To evaluate the characteristics of inference task workloads, AdaptiveLLM establishes a black-box Machine Learning (ML) model through an operator-level computational complexity analysis to predict the overhead of tensor recomputation. It also predicts the overhead of tensor swapping by conducting a fine-grained analysis of KV Cache memory usage. For the adaptive selection of offloading strategies, AdaptiveLLM designs a cost-aware memory optimization strategy specifically for the pre-emption scheduling phase. When GPU memory is insufficient, it opts for the offloading approach with a lower overhead. For the initiation scheduling phase, it devises a fairness-based user-request scheduling strategy. When GPU memory is available, it schedules more user requests in accordance with the principle of fairness. Experimental results indicate that, compared with currently widely used LLM inference benchmark frameworks, AdaptiveLLM achieves an overall increase in throughput while reducing the average weighted turnaround time, thereby realizing fair scheduling.

Key words: Large Language Models (LLM), inference, tensor swapping, tensor recomputation, throughput, fairness

摘要：

大语言模型(LLM)在多种自然语言处理任务中展现出卓越性能。然而, LLM拥有极高的参数规模, 使得有限的GPU内存容量成为推理任务的性能瓶颈。为此, 面向LLM推理服务场景提出AdaptiveLLM, 根据推理任务负载特征, 在张量交换和张量重算中实现卸载策略的自适应选择。为了评估推理任务负载特征, AdaptiveLLM通过算子粒度计算复杂度分析建立黑盒机器学习(ML)模型, 实现张量重算开销预测, 通过细粒度KV Cache内存占用分析实现张量交换开销预测。为了进行卸载策略的自适应选择, AdaptiveLLM针对抢占调度阶段设计一种基于开销感知的内存优化策略, 在GPU内存不足时选择开销较小的卸载方式。同时, 针对启动调度阶段设计一种基于公平性的用户请求调度策略, 在GPU内存空余时基于公平性原则调度更多的用户请求。实验结果表明, 相比于当前广泛使用的LLM推理基准框架, AdaptiveLLM实现了整体吞吐率的提升, 同时降低了平均带权周转时间, 实现了公平调度。

关键词: 大语言模型, 推理, 张量交换, 张量重算, 吞吐率, 公平性