Performance Characterization of LLM Inference under Limited GPU Resources

Abstract

With the widespread adoption of large language models (LLMs), the demand for graphics processing units (GPUs)-essential for accelerating LLM training and inference- has increased significantly. This has led to rising procurement and operation costs, imposing a significant financial burden on research institutions and industry. Efficient utilization of GPU resources has thus become a critical challenge. In this study, we investigate strategies to maximize resource efficiency by hosting multiple models on a single GPU rather than dedicating each GPU to a single model. We examined various GPU resource partitioning approaches to improve the utilization of limited GPU resources. Specifically, we compared two resource allocation methods for concurrently executing two models on a single GPU: using vLLM, a high-performance LLM inference framework, and using NVIDIA Multi-Instance GPU (MIG). The results demonstrated that the MIG configuration increased total throughput by approximately 700-950 tokens/s compared with vLLM-only execution, suggesting that partitioning a GPU into independent MIG instances can improve throughput for concurrent model execution.