Validity-Aware Multi-Level Autotuning for Efficient Tensor Program Generation

Abstract

The growing complexity of deep learning models necessitates specialized hardware and software optimizations, across diverse computing platforms, spanning from general-purpose CPUs and GPUs to domain-specific accelerators. While machine learning-based autotuning methods have emerged as a promising solution to reduce manual effort, existing template-based and template-free approaches often suffer from prolonged tuning times due to the extensive profiling of invalid configurations, which may result in runtime errors. To address this inefficiency, we propose MOCHA, a multi-level optimization with compiler for hardware acceleration framework designed for robustness and generality. MOCHA introduces two key ideas: (1) a validity prediction model to filter out invalid configurations prior to expensive hardware profiling, and (2) an advanced performance prediction model that leverages hidden features extracted during the compilation process. We evaluated MOCHA on two distinct hardware backends: a customized VTA accelerator, and an NVIDIA T4 GPU. Experimental results show that MOCHA on the VTA achieves equivalent performance improvements using only 12.3% of the samples required by a TVM-like approach and reduces the invalidity ratio during autotuning by an average of 66.4%. Similarly, on the NVIDIA T4 GPU, MOCHA demonstrates its superior efficiency by reaching the performance plateau using only 87.1% of the samples required by the baseline TVM autotuner while reducing the invalidity ratio during autotuning by 62.2%. These results highlight the potential of MOCHA to enhance autotuning efficiency and robustness across diverse hardware platforms by effectively filtering out invalid configurations and focusing the search on high-performance regions.