Thibaud Ardoin

Prompt Compression via Activation Aggregation

structure in the activation space of LLMs is revealed: (i) mid-layer representations transfer meaningfully to early layers, suggesting a degree of cross-layer compatibility in how information is encoded; (ii) a single activation vector encodes a quantifiable and recoverable amount of semantic information; (iii) a weig...