TenProf: A Tensor-Centric Profiler for Deep Learning Workload Analysis and Optimization

Abstract

Deep learning (DL) workloads span multiple layers of abstraction, from high-level Python scripts to intermediate C/C++ tensor libraries and low-level GPU compute kernels. While this layered design enables flexibility, it also obscures the root causes of inefficiencies, leaving GPUs underutilized. Profilers are essential for exposing such inefficiencies, yet existing tools fall short of explaining how high-level DL semantics translate into low-level GPU kernel and hardware behavior. They either offer DL operator-level statistics (e.g., for a convolution operator) with coarse-grained metrics such as execution time and memory usage, or focus exclusively on individual GPU kernels without linking them to DL semantics. As a result, true performance bottlenecks remain hard to diagnose and fix. We present TenProf, the first tensor-centric GPU profiler for PyTorch-based DL workloads. TenProf systematically bridges low-level hardware stall metrics with high-level DL semantics (particularly tensor transformations) through three key tensor-centric analyses: (i) tracing tensor transformations and modeling them as a forest, (ii) attributing GPU memory accesses to both base and view tensors, and (iii) assessing tensor access efficiency via hardware stall attribution. By unifying insights across layers, TenProf reveals how tensor accesses and transformations directly impact performance. It operates on unmodified program binaries, introduces moderate overhead, and generates interpretable profiles that guide actionable optimizations. Evaluation across several real-world DL applications shows that TenProf uncovers inefficiencies such as non-contiguous memory accesses, where resolving them requires only minor code changes yet yields non-trivial speedups. Several optimizations guided by TenProf have already been validated and upstreamed by application developers, demonstrating its practical value in production settings.