Corentin Kervadec

Tracing Computation Density in LLMs

The s-Trace method is introduced to efficiently estimate a subgraph of size s that approximates a full model output, and the amount of necessary computation per input correlates with model uncertainty, and that sparser subgraphs encode shallow statistics, such as unigram frequency.