Jun-Ping Zhao

LMTracer: Fine-Grained and Real-Time Performance Profiling for Production LLM Systems

Training and serving large language models (LLMs) has become a core business for AI providers. To ensure a high-quality user experience while optimizing infrastructure costs, providers need to closely monitor the performance of LLM executions in production. However, existing performance profiling tools fall short in th...