APPLICATION-LEVEL PERFORMANCE EVALUATION OF A MID-SCALE HPC SYSTEM FOR AI WORKLOADS
TL;DR
The study demonstrates that mid-range GPU infrastructure can effectively support modern AI applications, and provides practical deployment guidelines and demonstrate that mid-scale HPC systems can serve as a foundation for scalable and cost-effective AI infrastructure.
Abstract
This paper presents an application-level performance evaluation of a mid-scale high-performance computing (HPC) system, focusing on real-world AI workloads. Unlike traditional benchmarking studies that focus on hardware characteristics, this work evaluates practical performance using large language model (LLM) inference, embedding generation, and CUDA-based workloads. The study demonstrates that mid-range GPU infrastructure can effectively support modern AI applications, achieving up to 39.6 tokens per second for LLM inference, low latency (25-31 ms time-to-first-token), and high embedding throughput exceeding 56,000 sentences per second. The results further highlight the importance of software optimization techniques, including mixed precision, quantization, and optimized inference frameworks. A key contribution of this work is the evaluation of Kazakh language models, showing that language-specific models significantly outperform large multilingual models on domain-specific tasks. The findings provide practical deployment guidelines and demonstrate that mid-scale HPC systems can serve as a foundation for scalable and cost-effective AI infrastructure.