Typed Template Fuzzing

Abstract

Fuzzing finds bugs by testing software behavior with high volumes of input. To create semantically valid inputs for different targets, traditional fuzzers require significant engineering effort. LLM-based fuzzing approaches can be adapted to create inputs for a wide range of software but introduce new issues: every test needs an LLM query, sacrificing the speed of traditional fuzzing. We introduce Typed Template Fuzzing (TTF). It uses LLMs in a preprocessing step to create templates which encode semantic information of seed files. These templates enable semantically-aware mutations without the need for a parser, type-checker, or further LLM queries. We have implemented TTF in our tool Typos. Using Typos we have run a five-month fuzzing campaign, finding 97 previously undiscovered bugs. Typos achieves a 94.7% rate of valid inputs across a wide range of formats including SMT-LIB2, C, C++, Go, Rust, SystemVerilog, Scala, Java, and domain-specific libraries such as PyTorch and Qiskit.