Yaseen M. Osman

Activation-Based Active Learning for In-Context Learning: Challenges and Insights

A negative result is found: MLP and embedding layer outputs, viewed through the lenses of massive activations or the first four moments, do not correlate with example quality or task performance, suggesting that methods like Sparse Autoencoders (SAEs) may be a promising future direction.